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1.0 EXECUTIVE SUMMARY 


Technology is changing the way we live and learn. Large-scale social networks, interactive content, and 
ubiquitous mobile access have emerged as driving factors in the evolution of education and training. 
Enterprise analytics now allows for new ways to assess the effectiveness of training content and 
instructional strategies for different learners, and to understand organizational trends through access to 
large volumes of education data. 


Just as the increasing globalization of society underscores the importance of cross-cultural 
communication and understanding, the growth of technology enablers like cloud computing and the 
Internet of Things has increased the need for interoperability and cross-platform communication 
throughout the Department of Defense (DoD). The DoD’s Advanced Distributed Learning (ADL) Initiative 
is designing a framework of commercial standards, technical specifications, and business rules to enable 
plug and play interoperability of learning technologies. The Total Learning Architecture (TLA) framework 
will allow education and training products to interoperate with each other, with existing learning 
support systems, and with other DoD systems. 


The purpose of this report is to guide Command decisions toward a shared vision and end-state that 
benefits every warfighter individually and the entire DoD holistically. By necessity, the focus of DoD’s 
education and training community will evolve from simple compliance to a game-changing approach 
centered on innovation. Thus, measures of effectiveness will shift from inputs to outcomes. 


KEY TAKEAWAYS AND RECOMMENDATIONS 


Data is a critical asset that enables effective decision making. The TLA data strategy provides a common 
set of goals and objectives across DoD’s education and training community to ensure data are used 
effectively. This overarching strategy will ensure that all data resources are positioned in a way that they 
can be used, shared, and moved efficiently across the organization. This report describes the four pillars 
of the TLA data strategy. Each revolves around commercially available standards, and while they will 
continue to evolve, DoD education and training communities are urged to adopt and employ them now. 


These commercial standards describe the data within the four pillars of the TLA data strategy: 


e [EEE P9274.1 Experience API (xAPI) 2.0 — Learning activity tracking uses the xAPI to capture learning 
activity streams. The xAPI standard also includes xAPI profiles such as cmi5 and the TLA’s Master 
Object Model. xAPI 2.0 is targeted for approval in 2020. 


e IEEE 1484.12.1 Learner Object Metadata 2.0 — Descriptions of learning activities and their 
associated content are stored in the TLA’s Experience Index and use a modified version of the 
Learning Resource Metadata Initiative standard. A draft standard is being submitted for finalization 
in early 2020. 


e IEEE 1484.20.1 Reusable Competency Definitions - The definition of a competency, the relationship 
to other competencies, and the alignment of evidence to help measure proficiency of the 
competency, are included in this standard. This standard is expected for approval in 2020. 


e IEEE 1484.2 Interoperable Learning Records or IMS Global Comprehensive Learner Record - 
Learner profile standards do not currently meet all TLA requirements. These new standards are 
actively being developed and modified based on input from numerous industry groups and 
associations. 
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2.0 INTRODUCTION 


The TLA is a project by the ADL Initiative to develop a set of technical specifications, standards, and 
policy guidance that define a uniform approach for integrating current and emerging learning 
technologies into a future learning ecosystem. 


Throughout their careers, DoD personnel are educated and trained by various organizations, each using 
their own IT systems and business processes. Typically, these systems are developed and implemented 
independently, without coordination, causing duplication in function and stovepiping of the data 
maintained. Many of these systems have site- or agency-specific models for capturing data, or their own 
proprietary data repositories. Additionally, data transport, control, management, governance, and 
ownership are not easily compatible or interoperable across network boundaries. Therefore, there is 
now large-scale duplication of data and a lack of interoperability, transparency, and effective 
management to ensure DoD-wide data quality, availability, integrity, security and usability. 


The TLA vision recognizes that learning often occurs outside of formal education and training. In the 
current DoD environment, different tools, technologies, and platforms are used to track the knowledge, 
skills, attitudes, and other capabilities (KSAOs) acquired and certified through formal education and 
training. But the digital world around us creates even more opportunities to apply informal and non- 
workplace learning resources accessible from home, through social activities, and through alternative 
types of structured learning experiences. At the heart of the TLA concept is the learner and the 
associated policies, technical standards, and capabilities required to track all education and training 
experiences across the continuum of learning. 


The TLA project started in 2016 with the strategic vision of establishing a common data strategy across 
the education and training industry that enables lifelong learning. This goal required a multi-faceted 
understanding of the tools, technologies, modalities, and learning science used to support education 
and training within different communities of practice. The 2019 research refined and hardened the 
architecture, defined a standards-based data strategy, and established a 24/7 Reference 
Implementation as a shared resource to support test and evaluation of other DoD modernization efforts 


This report describes and summarizes the research performed by the ADL Initiative in 2019 within the 
TLA research portfolio. It includes four appendices that detail the TLA’s latest functional requirements, 
recommended draft standards, an initial architecture that conforms to the DoD Architectural 
Framework (DoDAF), and a Software System Design Document built around the 2019 TLA Reference 
Implementation. The target audience includes senior DoD leaders, educational institutions, and program 
managers of DoD training systems. The ADL Initiative invites review and feedback by technical 
personnel, subject matter experts, developers, academia, and other organizations interested in the 
interoperability of data related to lifelong learning. 


2.1 Problem Statement 


In the early years of Distributed Learning (DL), instructional content was constrained to online courses 
managed by a Learning Management System (LMS). The LMS controlled all aspects of the online course 
including the sequencing and delivery of content, tracking learner progress, managing learner records, 
and reporting. The Shareable Content Object Reference Model (SCORM) was created in the early 2000s 
to standardize the packaging, launch, and performance tracking of digital learning content so that it was 
reusable across different LMS platforms. SCORM helped revolutionize the way education and training is 
delivered, but it is limited in its ability to meet the needs of next generation learning content. 
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Today we understand that learning takes place both inside and outside the classroom. The digital world 
around us creates a web of learning resources that are accessible in the workplace, from home, through 
social circles, and on a growing variety of media and devices. The exponential growth of data generated 
by these systems has the potential to enable better insights while reducing costs through continuous 
process improvement across all education and training activities. Because work experiences are a 
significant part of learning, capturing and managing these learning experiences provides better insight 
into what an individual knows and what more they need to learn to support their current job. 


The technology available is constantly changing and the next generation of learning activities cannot be 
defined within the context of a single LMS. In today’s world, an individual’s lifelong learning continuum 
is distributed across numerous technology platforms that use different instructional methodologies and 
learning activities. The future learning ecosystem will be defined by personalized and competency-based 
learning environments that rely on the availability and accessibility of learner data across organizational 
and institutional boundaries throughout an individual’s life. 


To address these trends, many DoD components are undergoing their own modernization efforts, with 
direct support from the ADL Initiative. This is exemplified in the work being performed with the Office of 
the Undersecretary of Defense for Intelligence (OUSD(I)) on their Talent Development Toolkit? (TDT) 
project. Among other things, this project addresses the limitations faced by the Intelligence Community 
in exchanging learner-related data among 17 different organizations. 


Table 1 shows the high-level challenges such organizations face with the current factory model of 
education that is prevalent across DoD. The table summarizes how these problems manifest in the 
status quo and describes how TLA-enabled solutions can move DoD toward a desired end-state. 


Table 1. Learning Modernization Challenges. 
Status Quo Desired End-State 


STOVEPIPED: Most learning experiences are INTEGRATED: Learning experiences are integrated into a 
disconnected from one another; events lack cohesion | cohesive, career-long learning continuum by exchanging 
and treat learners as “blank slates.” learning data across systems. 


INEFFICIENT: Most education and training is “one- PERSONALIZED: Learning is optimized by delivering the 
size-fits-all,” which means top students waste time right education, training, and developmental experiences, 
and bottom students fail to master all content. at the right time, and in the right ways. 


WEAK DATA: Education, training, and personnel DATA-DRIVEN: Richer data enable talent management 
management systems rely on spotty data, which are | and readiness analyses—more effectively managing the 
often locked in unreachable data silos. larger system via evidence-based methods. 


CONVOLUTED: Operational goals and outcomes RESPONSIVE: More traceable (automated and data- 
generally lack direct connections to education and driven) connections with operations make education and 
training offerings, reducing agility and traceability. training more responsive and accountable. 


MONOLITHIC: Stovepiped systems have limited DECENTRALIZED: Interconnected, hardware-agnostic 
capacity for interoperability and require staff and capabilities that don’t require major investments in new 
new physical infrastructure to operate at larger scale. | servers and other physical infrastructure. 


INCOMPLETE: Current systems only document COMPREHENSIVE: Captures data from informal and 
formal education and training experiences available | unstructured learning resources and content, available 
through academic institutions and in the workplace. | froma growing variety of devices and providers. 


1 Talent Development Toolkit Requirements and Architecture Study - 
https://www.adInet.gov/assets/uploads/TDT%20Report.pdf 
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2.2 The TLA Vision and Objective End-State 


Ultimately, the purpose of education and training in the DoD is to provide capable manpower. This 
involves a variety of processes to determine workforce skill requirements, design learning systems 
aligned with those requirements, and develop the physical infrastructure, personnel, and technology 
resources to deliver the education and training. Maintaining this learning pipeline also requires 
continual evaluation of the efficacy of the learning products’. 


The key to managing lifelong learning data within the TLA is the interoperability afforded through the 
technical standards, specifications, and practices that underpin an integrated data strategy. The strategy 
is necessary to provide the semantic interoperability required for enterprise-level analysis and decision 
support. Data-driven decisions are enabled through enterprise-level analyses of learning data, 
supporting the continual refinement of manpower skillset definitions and the creation, selection, and 
maintenance of learning activities necessary to achieve proficiency. These also support the delivery of 
learning solutions required to acquire and maintain personnel with these skillsets. 


Technical specifications and standards allow different learning resources to communicate with each 
other using a common, agreed-upon “language.” These standards establish consistent protocols that can 
be universally understood and adopted by any component within the TLA to exchange data about 
learners, activities, and experiences. These underpinnings tie together the wide range of learning 
resources an individual will encounter and enable the sharing of this information with other systems and 
platforms across institutional boundaries. 


From a technical perspective, the desired end-state is the DoD-wide adoption of TLA standards, 
specifications, and policies that enable an interoperable fabric of learning data, federated within and 
across commands, agencies, and departments. As shown in Figure 1, the TLA is built around a four- 
pillared data strategy: 


e Experience Index of Registered Activities (e.g., 
Enterprise Course Catalog); 


Learning Record Store 
e Enterprise Learner Records (e.g., data on - Learning Event Tracking 


experiences, credentials, career trajectory); - Paradata 


e Competency Frameworks (e.g., the common 
definition of jobs, personnel, and learning 
outcomes); and 


e Learning Profile (e.g., tactical streaming data 
on learner performance). 


Beyond the technical interoperability standards Figure 1. Integrated Data Strategy. The TLA relies on 


i i i ; | data types that are stored within the “data 
for integrating tools, services, and devices, TLA four genera 
° siasreanie Ohana, lake” of a single TLA-compliant enclave. Core TLA 


research includes evaluations of shared services publish or subscribe to data streams that 
vocabularies, linked data, business rules, transform data into other meaningful information. 
governance, and a data strategy to tie them all 


together. 


? Instructional Systems Design, Systems Approach to Training (ISD/SAT) Process in Joint and Service instructions: MIL- 
HDBK-29612, NAVMC 1553.1, NAVEDTRA 14300, AR 350-70, and AFM 36-2234 
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Federating these data between different DoD components and their associated networks will enable 
enterprise-level services, including semantic interoperability, global discoverability, and security. It will 
provide an end-to-end system that is integrated, personalized, data driven, and responsive to changes in 
the threat and technical environment. This approach decouples the various learning functions typically 
included as part of a traditional LMS into composable data and services that address learning 
experiences across the continuum of learning and connect at-scale to data across the enterprise. This 
vision transitions the DoD from legacy, fragmented learning systems tied together with human driven 
processes into dynamically interoperable digital systems that provide the foundation for future 
improvement consistent with the DoD machine-learning strategy. 


The impact of the TLA data strategy and its associated Information Technology upgrades enables the 
development of a flexible, modular system to optimize the time, cost, and effectiveness of learning 
delivery and analysis. The TLA’s service-oriented approach results in interchangeable software services 
and databases integrated to create an efficient learning environment for the user. TLA components 
include specifications, software systems, tools, and services encapsulated into a Reference 
Implementation. 


The TLA research portfolio enables these capabilities by researching, developing, and promulgating the 
appropriate policies, specifications, and standards, but also by investing in key component technologies 
and interfacing with DoD stakeholders. 


2.3 Synopsis of Previous Research 


The ADL Initiative began development of the TLA in 2016. This work advanced to empirical testing in 
2017? which helped solidify the overall TLA conceptual design. The TLA Reference Implementation was 
developed to evaluate various software services, technical components, and learning activities, all of 
which exchanged data using an initial set of commercial standards. This allowed for a 2017 technical 
evaluation, managed by the Institute for Defense Analysis. 


In that 2017 evaluation, a panel of 54 experts participated in a Delphi-style feedback process that 
captured detailed reactions to the TLA’s perceived value, recommended technical standards to research, 
identified gaps, and made suggestions for increasing the likelihood of its future use. These insights 
informed a revision of the TLA concept, including its APIs, data models, specifications, and standards. 


In 2018, the TLA Reference Implementation was structured to assess the practical functionality of 
different commercial standards and their applicability to DoD requirements. A 2018 TLA Test and 
Demonstration focused on the integration of different learning activities into a competency-based 
program of instruction using varied instructional modalities. A competency framework was created to 
house the competencies related to Combat Awareness, Combat Profiling, and Combat Tracking which 
are all part of the Joint Staff’s Combat Observation and Decision-Making in Irregular and Ambiguous 
Conflicts (CODIAC) course. 


The 2018 TLA Test and Demonstration was held to test general system functionality, evaluate and refine 
candidate standards, and measure cost factors for instrumenting content. Detailed information about 
this event and the lessons learned are provided in the 2018 TLA Report’. 


3 TLA Design-Based Research Approach - https://www.adInet.gov/assets/uploads/2017%20--- 
%20%28lITSEC%29%20Gallagher%20et%20al.%20-%20Total%20Learning%20Architecture%20Development.pdf 
4 2018 Reference Implementation Specifications and Standards - https://apps.dtic.mil/dtic/tr/fulltext/u2/1077398.pdf 
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The 2018 event exposed technical gaps and identified architectural concerns about state management 
and the point-to-point messaging infrastructure. The 2018 components also used closely coupled 
datastores that did not accurately reflect the data streaming architectures currently in use by DoD. 


For the 2018 event, learning activities were instrumented with the Experience API (xAPI). Upon analysis 
of the data, inconsistencies with the quality and uniformity were identified, including invalid statements 
and data formatting from 3™ party systems. The 2018 dataset contained invalid xAPI statements due to 
empty object identifiers, choice interaction components having an empty string for an ID, or an invalid 
statement ID within a reference object. Learner data collected from 3" party systems were also stored 
in formats that could not be used by other TLA components. 


Key factors hindering the quality of the dataset included: 


e Inconsistency in actor, verb, object naming patterns — Many statements were missing fields, such as 
the object activity definition. Non-unique IDs and/or other fields were also being used as unique 
identifiers. It is essential to have at least one unique identifier for each entity type, and that 
uniqueness must be maintained. 


e = Hidden links in recommender and competency management system — A loss of traceability into 
“black box” components made it difficult to reconstruct evidentiary chains. Data generation 
processes designed in relation to the recommender were update-in-place, including the Experience 
Index. This resulted in the loss of valuable time-series data. 


e Poorly formatted objects — Different objects followed different schemes for capitalization, space 
padding between fields, and field order. There was a significant amount of inconsistency across 
statements of the same types. 


e Inconsistent labeling of competency levels — Internal competency data did not match external 
documentation, and learning resources described different relationships between these terms. The 
Experience Index contained undefined and unnecessarily excessive data. Additionally, the original 
design did not adequately protect identifier uniqueness and required an update. 


The 2019 TLA research built upon lessons learned from 2017 and 2018, resulting in a complete redesign 
of the Reference Implementation. Integrated data services were enabled through an improved data and 
communication architecture built around the Kafka data streaming platform. This meant selecting 
composable data models, enhancing portability of devices and activities for generating or using learning 
data, designing scalable services for communicating and sharing these data, and extending control over 
the data to address privacy and cybersecurity concerns. 


As the TLA progresses toward an Initial Operational Capability in 2021, near-term research goals are 
focused on establishing tools, technologies, and policies that maximize adoption of the API (xAPI) and 
reduce barriers to entry (e.g. cybersecurity compliance) across the DoD. This foundation enables the 
collection of learner-related data to support enterprise learner analytics. Longer term research focuses 
on integrating machine learning and artificial intelligence with this analytic capability to build adaptive 
instructional systems for career management within the human capital supply chain. 


2.4 A Decentralized Approach 


The TLA vision features non-monolithic, decentralized solutions to the big-data challenges faced by 
DoD’s education and training enterprise. There is no TLA mainframe, no single data repository, no TLA 
bureaucracy to manage its systems and services. 
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By providing a set of executable policies, specifications and standards for data collection and sharing 
(essentially an integrated data strategy for DoD education and training), the TLA enables the 
modernization of DoD learning systems, and their ultimate integration within the agency’s enterprise 
management business processes. In most cases, this can be accomplished with phased upgrades or 
augmentations to current systems, rather than costly replacements and new-systems acquisition. 


The desired transition is from fragmented legacy systems tied together with human driven processes, to 
a dynamically interoperable digital system that supports automation and is responsive to changes in the 
threat and technical environment. 


3.0 RESEARCH INTO COMMERCIAL STANDARDS 


In 2018, a wide range of commercially available standards were evaluated on their ability to support 
different aspects of an integrated data strategy. Research was performed through different ‘lenses’ to 
identify how standards and specifications might be used at various stages of a career trajectory, and 
across institutional boundaries. This research highlighted candidate standards that warranted further 
evaluation to determine their ability to meet future DoD requirements. 


To mature the data strategy, 2019 efforts included participation in multiple standards initiatives and 
industry working groups, and collaboration with other DoD stakeholders. Vendors under contract with 
ADL Initiative were required to provide technical reports that detailed how their tools, technologies, and 
capabilities addressed specific elements of the data strategy. These reports provided insight into the 
strengths, weaknesses, commonalities, and challenges of relevant standards and technical approaches 
being used to support education and training across the DoD. TLA standards development in 2019 fell 
into five broad categories: 


e Overarching TLA specification (e.g., business rules, implementation requirements, cybersecurity, 
protection of personally identifiable information (PII)) 

e Competency-based learning 

e = Learning activity tracking 

e Learning activity and resources metadata 

e = Learner records 


The evaluation of candidate standards started with the development of use cases derived from 
literature reviews, interviews with subject-matter experts, and collaboration with stakeholders. Figure 2 
shows the iterative method used to develop design hypotheses and prototype the experimental test bed 
to evaluate candidate standards. 


Concept Maps v= 
Heuristic Evaluation <> A = & 
Literature Review — 
Requirements Prototype 
Test Analysis 


Analysis Analysis 


Figure 2. TLA Research Methodology. Iterative processes were used to establish an integrated TLA data strategy. 
The TLA technical team was comprised of engineers, learning scientists, designers, and developers. Use cases were 
developed to guide experiments that validate key concepts. 
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The research team used this approach to define requirements and develop models? describing the 
physical, functional, semantic, and operational architecture required to evaluate the adequacy of each 
standard. This approach also allowed the team to measure the impact each standard has on other 
related standards and overall system performance. Each standard was verified iteratively and mapped 
to the features and capabilities of each TLA system node and its associated interfaces. 


TLA technical specifications and standards allow different learning resources to communicate with each 
other using a common language. These standards serve as building blocks for lifelong learning by 
establishing consistent protocols that can be universally understood and adopted by any new learning 
capability, enabling data exchange about learners, activities, and experiences. This section describes the 
baseline TLA standards that were defined through this design/build/test process, as well as stakeholder 
engagements used to validate them. 


3.1 Overarching TLA Specification 


TLA standards follow a model analogous to the SCORM specification, which allows the standardization 
of packaging, launch, and performance tracking of digital content so that it is reusable across different 
LMS platforms. SCORM is comprised of multiple commercial standards from the Institute of Electrical 
and Electronics Engineers (IEEE) Learning Technology Standards Committee (LTSC), IMS Global, and the 
Aviation Industry Computer-Based Training Committee (AICC). The SCORM specification pulls these 
standards together into a Reference Model that dictates how they interoperate. Similarly, the ADL 
Initiative anticipates that the overarching TLA specification will pull together a multitude of other 
accepted standards to dictate how systems, platforms, and technologies will interoperate within the 
future learning ecosystem. 


Figure 3 shows how this specification might support the modernization of the Defense Health Agency 
(DHA) education and training infrastructure, linking commercial standards, cybersecurity policies, and 
business rules together to support the Military Healthcare System (MHS) Continuum of Learning. 


Systems Approach to MHS Learning Continuum 


People Process Technology 


Trainin Eynerience Professional 
4 Development 


Figure 3. Systems Approach to the Military Healthcare System’s Learning Continuum. The DHA relies upon the 
unified performance of people, processes, and technology components across the DoD enterprise. 


* Model Based Systems Engineering (MBSE) methods — Systems Engineering Body of Knowledge 2.1 
https://www.sebokwiki.org/wiki/Guide_to_the_Systems_Engineering_Body_of_Knowledge_(SEBokK) 
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The ADL Initiative supported the Naval Air Warfare Center, Training Systems Division (NAWCTSD) in their 
development of requirements to support the DHA modernization. These requirements are not unique to 
DHA but illustrate the types of systems and technologies at play across the DoD. DHA uses various 
learning technologies (e.g., computer-based training, classroom instruction, intelligent tutoring, patient 
simulators, continuing education) across a widely distributed collection of facilities, schoolhouses, and 
training sites. This work validated the requirements defined within the OUSD(I) TDT project for an 
overarching TLA specification that ties commercial standards, policy guidance, governance strategies, 
and business rules to facilitate interoperability across the enterprise. 


Recognizing the value of this approach, the IEEE began updating the IEEE 1484.1 Learning Technology 
Systems Architecture (LTSA) standard. A study group® was formed to use the U.S. Department of 
Education’s Common Education Data Standards (CEDS) Normalized Data Schema (NDS), coupled with 
other IEEE standards and other TLA standards and specifications to create a Conceptual Model for 
Learning Technology Standards (CMA4LTS). This work will eventually result in an updated IEEE LTSA 
standard which outlines an architectural framework of systems, components, functions, and their 
related data standards. 


Figure 4 shows how different components of the TLA map to different commercial standards under the 
CMALTS approach. This effort is relatively new but has broad support throughout the community. It has 
potential to become the overarching TLA specification, but additional work is needed to add other 
standards to support Federated Identity, Credentials, and Access Management (FICAM), Zero Trust 
Networks, and cybersecurity compliance using NIST 800 and other risk management controls. 
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Figure 4. Federated Data Approaches within the IEEE CMI4LTS Study Group. This illustration shows where different 
commercial standards are applied across a range of enterprise learning technologies. 


® IEEE Conceptual Model for Learning Technology Standards (CM4LTS) Study Group Charter 
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3.2 Competency-Based Learning 


Competency-based learning represents a transition from curricula focused on abstract knowledge and 
pursuit of certificates to curricula built around authoritative learner performance and proficiency 
indicators. Competencies describe specifically what people need to do to be effective in their roles, and 
clearly establish how their roles relate to organizational goals and success. Each job has its own set of 
competency requirements. Proficiency is another critical concept that requires relevant, trusted data as 
evidence of mastery of specific skill requirements. 


Competency management requires the generation of rich, traceable data about learning experiences, 
how they relate to skill proficiency, and the KSAOs that individuals need to do their job. A Competency 
Framework is a structure for defining the KSAOs required to do a job. Competency Frameworks are 
widely used in business for defining and assessing competencies within organizations for successful job 
performance. There are numerous competency frameworks available and numerous specifications that 
drive them. Some of these are included later in this section. 
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Figure 5. IEEE 1484.20.1 Reusable Competency Definition. The RCD standard provides a mathematical formalism 
for defining competencies and describing the relationships among competencies within a Competency Framework. 
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Through 2019, the ADL Initiative technical staff continued working with the IEEE 1484.20.1 Reusable 
Competency Definition (RCD) study group. As shown in Figure 5, the RCD standard uses linked data to 
define all aspects of a competency including key performance indicators, formal assessments, and other 
measures of proficiency. The RCD standard is the mathematical underpinning of the TLA’s approach to 
competency-based learning. It provides a format for defining competencies and associating them with 
other competencies in the context of an overarching Competency Framework. 


Mathematically, the RCD must behave as a Directed Acyclic Graph (DAG). In graph theory, a DAG 
structure comprises nodes connected with edges. Each node in the graph corresponds to a competency 
object, and the edges define the relationship between them. Within the TLA, each RCD node includes a 
unique identifier that can be referenced by other TLA components using other TLA standards. For 
example, the Learning Resource Metadata Initiative’ (LRMI) and the Credential Transparency 
Description Language® (CTDL) both reference these identifiers to provide an alignment between learning 
activities, competencies, and awarded credentials. 


Competency models and frameworks exist in many different formats that are supported by different 
communities, including the Federal Government. The ADL Initiative, through the development of 
authoring tools for the Competency and Skills System (CaSS), worked with the Air Force A3J 
Competencies Division and participated in working groups across industry, academia, and government 
to better understand how competencies are derived. A report titled The Competency Framework 
Development Process?’ was collaboratively authored to document these processes within the Air Force 
and across different industries. These projects provide a good understanding of the complexities and 
differences in how competencies are derived across different DoD organizations. 


The CaSS authoring tools project follows this development process and is creating intuitive workflows 
for importing and exporting other Competency Frameworks such as OpenSALT, the Achievement 
Standards Network (ASN), the Competency and Academic Standards Exchange (CASE) specification, and 
Medbiquitous. CaSS also supports CTDL and the IMS Global OpenBadge specification. 


As shown in Figure 6, the TLA Reference Implementation uses CaSS to process and align evidence of 
learner performance against a Competency Framework. The management system asserts levels of 
proficiency for individuals or teams, allowing credentials to be awarded based on those assertions. 


Can 
Generate Ba aVi(e(-1alecKe)i peaked Competency | —N™) Assertion of 
Performance Manaee merit Competency 


SS May Align To | 


Thea Include Processes Evidence t 


As Defined By 
Content and/or 
Experience 


Figure 6. Relationship of Competency and Assertions. Learning activities generate evidence of performance that is 
processed by a Competency Management System. The Competency Management System aligns the evidence to the 
Competency Framework and asserts levels of proficiency for individuals or teams. Credentials may also be awarded 
based on assertions. 


(@olaisansve| 
Competency Credentials 
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7 Dublin Core, About LRMI - https://www.dublincore.org/about/Irmi/ 
8 Credential Transparency Description Language Handbook - http://credreg.net/ctdl/handbook 
? Currently going through the government acceptance process 
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Figure 7. Relationship of Competencies and the Squad Performance Model. At any given moment, dozens of 
competencies, measures of effectiveness, and measures of performance can be happening simultaneously. This is 
especially true in collective training where team competencies such as trust, confidence, and communication create 
different types of evidence than individual competencies. 


In another 2019 competency-oriented project, described in Figure 7, the ADL Initiative supported the 
Army’s design and development of a Squad Performance Model and a prototype capability to track and 
measure individual and team competencies. The Army’s Synthetic Training Environment — Experiential 
Learning (STEEL) project intends to develop a prototype capability that tracks and measures individual 
and team competencies across all Synthetic Training Environment (STE) Training Aids, Devices, 
Simulators and Simulations. These activities connect to STE Training Management Tools to communicate 
individual and team performance information to update the Squad Performance Model and report to 
the Army Training Management Capability that maintains individual and unit training records as part of 
the Army Training Information System (ATIS) program. 


Competency-based learning standards intersect with numerous other TLA standards. 


e IEEE 9274.1 xAPI is used to track learning experiences and make assertions that can be read by a 
Competency Management System. 

e LRMI is a metadata framework developed by Creative Commons and the Association of Educational 
Publishers for describing and tagging educational resources in web-based instruction and training. 
LRMI includes the ability to align educational resources to the competencies within the RCD 
Competency Framework. 

e IMS Global Open Badge is a technical specification and set of associated open source software to 
enable the creation of verifiable credentials across a broad spectrum of learning experiences. 
Badges are used to represent successful completion of key milestones within a program of 
instruction (e.g., successful completion of a learning activity). 

e The CTDLis a vocabulary comprised of terms that describe each credential. Credentials are related 
(linked) to other entities in the credentialing ecosystem such as assessments, learning opportunities, 
requirements, costs, and conceptual frameworks (e.g., competencies, classifications of occupations, 
and instructional programs). 
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3.3 Learning Activity Tracking 


DoD Instruction 1322.26 recommends the /EEE 9274.1 xAPI?° standard as the contemporary method for 
managing learner-performance data. During 2019, the xAPI specification entered the final stages of 
becoming a standard through the IEEE-LTSC"4, with xAP/ 2.0 targeted for final approval in 2020. The ADL 
Initiative established a DoD working group to identify challenges and roadblocks for adopting xAPI 
across the DoD enterprise. This group exposed several challenges associated with cybersecurity and the 
availability of DoD accredited Learning Record Store (LRS) solutions. Their findings were summarized in a 
paper titled Cybersecurity Strategies for Accrediting xAPI"2. 


The xAPI specifies a structure to describe learning experiences and defines how these descriptions can 
be exchanged electronically. The main components of xAPI are the data structure called Statements and 
the LRS data storage/retrieval capability. The xAPI specification has stringent requirements on the 
structure of these data and the capabilities of the LRS. Statements are data triples that use an Actor, a 
Verb, and an Object to describe any experience. Each statement also includes timestamps and unique, 
resolvable identifiers. The transport is HTTP/HTTPS with JavaScript Object Notation (JSON) payloads. 


Any enabled device can send xAPI statements, including tablets, phones, simulators, patient mannikins, 
and any number of other learning systems. When collecting data from these systems, organizations 
need to be able to shape their xAPI data, standardize it semantically, and identify unique patterns and 
contexts. The xAP! Profile Specification (IEEE 9274.2) offers a common way to document the vocabulary 
concepts, extensions, statement templates, and patterns that are required for xAPI to be implemented 
consistently across the spectrum of learning activities an individual will encounter. Upon the successful 
approval of the xAPI 2.0 specification, the IEEE working group will begin standardizing the xAPI Profile 
Specification. 


An xAPI Profile Server project was started in 2019 to provide the xAPI developer community with a 
platform for authoring, sharing, and validating xAPI profiles that conform to the specification. The xAPI 
Profile Server project also provides tools to support the management, discovery and governance for 
adding or modifying data elements and properties within each profile. One of the best-known examples 
of an application-specific xAPI Profile is the cmi5”? specification. The cmi5 specification defines a set of 
rules for importing, launching and tracking online courses using an LMS and xAPI. Technically, cmi5 is an 
xAPI Profile, which means it inherits the characteristics mandated by the xAPI specification, but cmi5 
also imposes additional requirements. These include interoperability rules for content launch, 
authentication, session management, reporting, and course structuring. The xAPI working group found 
that many DoD LMSs and authoring tools do not support cmi5, and others lack a cmi5 conformance test 
suite for validating adherence to the cmi5 specification. 


While the xAPI and xAPI Profile specifications are extensible and can be integrated into any activity used 
for education and training, the TLA Master Object Model (MOM) (IEEE 9274.3.1) normalizes the way 
xAPI evidence is reported from each learning activity to the TLA core (see Section 4.3). The MOM 
contextualizes how the evidence is gathered, and how the learning environment is organized. 


10 ADL Initiative GitHub - https://github.com/adInet/xAPI-Spec 

1 IEEE LTSC TAGXAPI - https://www.tagxapi.org/ 

?2 Cybersecurity Strategies for Accrediting xAPI - https://s3.amazonaws.com/amz.xcdsystem.com/44ECEE4F-033C-295C- 
BAE73278B7F9CA1D_abstract_File4313/PaperUpload_19308_0616074810.pdf 

13 The cmi5 Project - https://aicc.github.io/CMI-5_Spec_Current/ 
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Figure 8. The Learning Lifecycle. The MOM learning cycle is captured in the TLA DoDAF and outlined in the draft 
IEEE specification attached as Appendix B. This lifecycle accommodates deliberate learning (planned or required) 
and informal learning captured from casual interactions with instrumented media. 


As shown in Figure 8, the TLA MOM abstracts performance adjudication to edge systems by following 
the learning event lifecycle. Learning activities report success or failure using the cmi5 verbs, whereas 
lower level performance data may be kept on a local LRS. This enables the evaluation of trust in the 
evidence used to update competencies and credentials. It also provides reference points within 
collected performance data to filter and organize enterprise analytics within the decision support 
system. Most importantly, it normalizes the events necessary to trigger microservices to take actions 
without an overall state manager. This forms the decoupling and data segmentation approach used for 
LRS federation. 


Within the TLA MOM are embedded chains of evidence that ensure “learner centricity” is preserved 
across a wide range of learning activities or other TLA components. This establishes an auditable trail of 
demonstrated competency to back-up each awarded credential. It also preserves the digital signatures 
of those trusted to review and confer credentials. Since the TLA MOM allows for learner context 
generated by any number of locations in the federation, it also ensures resiliency to system changes 
over time. The draft IEEE 9274.3.1 TLA MOM standard is included in Appendix B. 


The xAPI standard intersects with numerous other TLA standards including: 


e The xAPI Profile Specification (IEEE 9274.2) provides a common vocabulary and identifies specific 
contexts and patterns using linked data. 

e The TLA MOM (IEEE 9274.3.1) represents a learner’s state in the context of a learning experience 
within a continuum of learning. 

e Thecmi5 specification is used to replicate the functionality resident with SCORM managed courses 
delivered using an LMS. 

e SCORM is currently being extended through the IEEE LTSC to maintain the DoD investment into 
SCORM conformant courses. It will continue to play a role until content can be migrated to other 
specifications (e.g., cmi5). 

e The ePUB3 standard for eBooks that use Personalized eBook for Learning (PeBL) extensions enables 
generically instrumented eBooks for learning. 
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3.4 Metadata about Learning Activities 


The 2018 TLA demonstration relied heavily on the LRMI metadata framework for describing and tagging 
educational resources. The LRMI is a metadata framework developed by Creative Commons (CC) and the 
Association of Educational Publishers (AEP). It is used for describing and tagging educational resources in 
web-based instruction and training. A primary benefit of the LRMI specification is its inclusion of an 
“AlignmentObject” to describe an alignment between a learning resource and a competency node in an 
RCD framework. 


2019 research continued the process of mapping the relevant attributes of existing metadata standards 
including LRMI, IEEE 1484.12.1 Learning Object Metadata** (LOM), Schema.org, and the Dublin Core 
Initiative. The ADL Initiative participated in a series of technical working group meetings that included 
the working groups from various metadata standards initiatives, industry partners, and other 
organizations working to harmonize across available standards. This work informed a draft metadata 
strategy that was initially created to support the migration of Adobe Flash content to an HTMLS friendly 
format. The draft metadata strategy merged the LOM standard with LRMI to support newer learning 
modalities (e.g., Augmented- and Virtual-Reality, instructor-led, serious games, simulations). 


Through the TLA working group’s subcommittee on metadata, ADL Initiative staff continued to evolve 
the metadata strategy based on the technical interchange with different DoD stakeholders. These 
discussions helped align the goals of the subcommittee with the DoD’s programmatic requirements 
across different modernization efforts. This stimulated the standards community to start the formal 
process of updating the IEEE 1484.12.1 LOM standard. The LRMI working group and the LOM working 
group started working together to formally merge LRMI and LOM into a LOM 2.0 standard. This effort is 
expected to be formalized into an IEEE study group in 2020. 


Based on this work, the 2019 TLA Reference Implementation expands the types of learning activities a 
learner may encounter by acknowledging the relationship between instructional content and the various 
instructional activities where that content is presented to the learner. The term “activity” describes the 
context of the work, such as simulation, LMS, eBook, or classroom lecture. The term “content” describes 
the digital artifacts used in the activity, such as a scenario, Sharable Content Object, checklist, or 
technical manual. 


Experience 
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Figure 9. Relationship between Experience, Activity, Content, and Course. Data elements segmented for 
configuration management and search performance and exported in a modified LRMI format. 


44 IEEE Standard for Learning Object Metadata - https://ieeexplore.ieee.org/document/1032843/ 
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The draft LOM 2.0 metadata strategy created in 2019 provides a broad framework for describing 
learning objects to facilitate search, evaluation, acquisition, and use. This relationship, shown in Figure 
9, is encapsulated into a learning experience aligned with a fragment of a Competency Framework that 
includes one or more competencies. 


A large component of the 2019 research involved defining the lifecycle of metadata across human- 
readable and machine-readable formats. While human-readable metadata is typically created during the 
development of instructional activities, the next generation of learning tools will automate the process 
of creating machine-readable metadata that uses paradata’ about how each activity and its content is 
being used across the enterprise. The Navigator for Integrated Learning Experiences (NILE) project?® 
studied this topic in detail and delivered detailed documentation describing how their algorithms 
automated this process. The NILE project found that many human-readable metadata formats such as 
Dublin Core and LOM were unusable by adaptive instructional systems. 


While standard search algorithms can find resources relevant to key terms and to a given competency if 
alignment data are present, this often requires manual labor. The greatest need for curating learning 
resources using machine learning and artificial intelligence is access to usage data. Within NILE, paradata 
can be used to automatically create or update metadata describing relevance, engagement, and 
efficacy. 


This is especially relevant to the DoD’s Enterprise Digital Learning Modernization initiative, led by the 
DoD Chief Management Office (CMO), as department-wide and/or service-specific governance 
strategies are developed to manage an integrated data strategy. In 2018, the CMO’s Reform 
Management Group issued a decision memo instructing DoD to develop a common catalog of education 
and training courses. The Enterprise Course Catalog (ECC) project intends to provide a globally 
searchable directory of DoD course listings and ancillary training resources, pulling metadata from local 
sources/catalogs. More than just another static course catalog, the ECC will federate numerous local 
course catalogs such that DoD users see one seamless interface to access course information at the time 
and place of need. 


ADL Initiative research in 2019 informed requirements for key features of this work including the ability 
to perform semantic search services across Federated Experience Indices at scale, managing paradata to 
continually update metadata, and an underlying messaging architecture that conforms to the DoD Chief 
Information Officer’s policy for Information Enterprise Architectures?’. 


Learning activity metadata intersects with other TLA standards and capabilities including: 


e Competency-based learning assertions in the form of IEEE 9274.1 xAPI statements use an Object ID 
that is included in the LRMI/LOM metadata for each activity. A Competency Management System 
uses this ID to pull metadata about the learning activity that generated the assertion. 

e The LRMI Educational Alignment field can be used to map each learning activity that the unique 
identifier included in each IEEE.1484.20.1 RCD node. 

e Metadata describing learning activities and their associated content is stored in the TLA Experience 
Index for use by other TLA components. 


45 Paradata describe data generated as a by-product of the data collection process. 

16 Navigator for Integrated Learning Experience - https://adInet.gov/projects/nile/ 

17 DoD Enterprise-level Architectures - https://dodcio.defense.gov/In-the-News/DoD-Information-Enterprise- 
Architecture/Transformation-Context/ 
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3.5 Learner Records 


The current way learner records are managed is insufficient for the evolving needs of instructors, 
learners, and organizations. Today, academic institutions use a transcript to record learners’ permanent 
academic records. Typically, a transcript only includes the most basic of information such as courses 
taken, dates, grades received, and degrees conferred from a formal academic institution. Transcripts 
offer little visibility into individuals’ past performance, such as what other instructors have noted about 
them, the informal or nonformal learning they’ve experienced, or their strengths, weaknesses, and 
individual needs. 


The ADL Initiative supported multiple projects that could enable more expansive DoD access to learner 
records. Described below, these projects are being led by the T3 Innovation Network (T3 Network), the 
American Association of Collegiate Registrars and Admissions Officers (AACRAO), the National 
Association of Student Personnel Administrators (NASPA), and the American Workforce Policy Advisory 
Board (AWPAB). In many instances, pilot projects are moving forward focused on academic, military, 
and workforce domains. These projects provide opportunities for ADL Initiative staff to engage and 
leverage lessons-learned and related solutions to ensure the military perspective is accurately and 
comprehensively represented. 


The persistent theme across these efforts is that current projects limit their focus to harmonizing 
credentials, over the distribution and optimization of credentials for jobs. Overemphasizing credential 
portability leaves them vulnerable to being devalued, and if an organization relies solely on the 
accrediting body there is no mechanism to review the chain of evidence for the credential’s conferral. 
Forensic analyses following any negative impacts on readiness are complicated, especially for 
credentials that require different learning modalities where behaviors imparted in traditional learning 
may not always be transferred. 


3.5.1 T3 Network 


Established through collaboration between the U.S. Chamber of Commerce Foundation and the Lumina 
Foundation, the T3 Network’? is leading eight pilot projects to establish an open, distributed, public- 
private data infrastructure that supports access and opportunity for American students and workers. 
The ADL Initiative is engaged in three of these pilot projects to represent the interests of DoD and garner 
lessons-learned for the TLA project. 


The three pilot projects are described below. 


e Data Standards Harmonization (PP1) — Designed to identify gaps in commonly used data standards, 
and work across projects and standards to improve and harmonize data standards. 


e Comprehensive Learner, Worker, Military Records (PP3) — Designed to identify gaps in standards to 
ensure that records are complete with their full range of competencies and credentials. 


e Competency Data Exchange (PP5) — Designed to develop business models to incentivize the sharing 
and distribution of competency frameworks and develop protocols and processes for sharing 
competencies across networks. 


18 U.S. Chamber of Commerce, The T3 Innovation Network - https://www.uschamberfoundation.org/t3-innovation 
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3.5.2 AACRAO & NASPA 


In 2017 AACRAO and NASPA completed Phase One’? development of a Comprehensive Learner Record 
(CLR) to meet the needs of a broad number of U.S. colleges and universities. The CLR is intended to 
capture, record, and communicate learning when and where it happens in a student’s higher education 
experience. In their Phase 2 report, Standardized Components for a Competency-Based Educational 
Record”’, the team described their success in: 1) scaling of CLRs in up to 150 colleges and universities; 2) 
standardizing the components of a competency-based education transcript/record; 3) addressing data 
integration issues; and 4) leveraging existing degree audit technologies to track progress toward learning 
outcomes. The ADL Initiative is monitoring this effort to apply lessons learned to DoD requirements. 


3.5.3 AWPAB 


In support of the National Council for the American Worker (NCAW), AWPAB is developing 
recommendations across four focus areas. The TLA’s interests are most aligned with the Data 
Transparency working group, which is focused on enabling a system for translating comprehensive 
education, training, and work experience to a record of transferable skills and credentials throughout a 
worker’s entire learning lifecycle. In September 2019, the Data Transparency working group released a 
report on Interoperable Learning Records”? (ILR), featuring key ILR terminology and a forward-looking 
ILR ecosystem vision. The ILR approach includes three objectives: 


e Create an ILR inventory (October 2019) 
e Convene an expert group to develop a project plan (December 2019) 
e Champion fast-track prototyping (2"4 Quarter 2020) 


This industry-focused effort largely defined learner records according to use-cases that were established 
to represent the learner and employers. These use-cases did not consider how adaptive instructional 
systems might use learner records. They were derived to represent how individual learners might use a 
CLR to manage career growth, or how employers might use learner records to support the human 
capital supply chain and workforce planning in the context of a Job Data Exchange”. 


The ADL Initiative continues to monitor this work through the close coupling of this initiative with the T3 
Network’s pilot projects. 


3.5.4 IMS Global Comprehensive Learner Record 


The IMS Global CLR”? standard is a promising format for an expanded, more data-rich academic 
transcript which captures detailed learning experiences and learner achievements such as prior learning, 
internships, experiential learning, coursework taken and completed, competencies, skills, and co- 
curricular achievements. The CLR standard lays the groundwork for contextualizing learning data at the 
institutional level and enables basic interoperability by allowing organizations to digitally maintain and 
transfer credentials throughout a learner’s career. 


19 Integration of Data Across Institutional Platforms to Create Comprehensive Learner Records - 
https://www.aacrao.org/docs/default-source/signature-initiative-docs/clr/data-integration-white-paper-9_2018.pdf 
20 Standardized Components for a Competency-Based Educational Record - https://www.aacrao.org/docs/default- 
source/signature-initiative-docs/clr/competency-based-educational-record-report-092019.pdf 

21 White Paper on Interoperable Learning Records - https://www.commerce.gov/sites/default/files/2019- 
O9/ILR_White_Paper_FINAL_EBOOK.pdf 

22 Job Data Exchange - https://www.uschamberfoundation.org/workforce-development/JDX 

23 Comprehensive Learner Record - http://www.imsglobal.org/activity/comprehensive-learner-record 
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The major components of the CLR include information about what the learner learned (e.g., 
achievements, their relationship to an educational framework, and relationship to other achievements), 
evidence of their achievement (e.g., assertion, endorsement claim, evidence (artifacts like text, media, 
and website that provide supporting evidence for the record), and verification), and personal 
information (e.g., address, identity, profile). The tangible result is a web-ready linked-data document 
that can be shared externally. It supports JavaScript Object Notation for Linked Data (JSON-LD), allowing 
it to interface with the data ecosystem of digital credentials. 


These capabilities only partially meet DoD requirements. The current CLR structure uses six core classes 
to define its data properties; 1) CLR, 2) Learner Profile, 3) Publisher Profile, 4) Assertions, 5) 
Achievement, and 6) Association. Five service interfaces help manage access to the relevant CLR 
components and can be used to share this data with other TLA components. The core components of 
the CLR are largely focused on achievements represented by credentials, their relationship to an 
educational framework, and their relationship to other achievements. There is a risk that without access 
to localized learner records where evidence of competency often resides, credentials could become 
conflated with the evidentiary chain required to support CBL. DoD requires access to an authoritative 
chain of evidence which will be stored across DoD in various LRS solutions. 


3.5.5 IEEE 1484.2 Integrated Learner Record (ILR) Study Group 


In June 2019, the IEEE LTSC obtained approval for the IEEE 
1484.2 Integrated Learner Record (ILR) study group to move W3C DID 
forward, with a goal to provide interoperability across 


W3C VC 


numerous learner record initiatives. This group is comprised 


of standards-focused members from the Department of T3 LWMLR 


ADL ELRR 
IEEE ILR Alignment Reference 
Implementation 


IMS OB 


Education, IMS Global, the Postsecondary Educational Subject to 
Standards 


Standards Council (PESC), and other related IEEE standards Making 
groups. The ILR Recommended Practices will build upon the es 
CEDS V8 Conceptual Data Model and provide best-practice 

guidance for self-sovereign Identity and Access Control IMS CLR 
Trust Networks, open ontology references, and verifiable 
assertions. 


Europass 


; ; Figure 10. IEEE 1484.2 Integrated Learner Record. 
This project started to address requirements set by the T3 The ILR provides interoperability across numerous 


Network and AWPAB, and to facilitate integration with learner record initiatives. 

other standards initiatives. As shown in Figure 10, the ILR 

intends to be compatible with different learner records that might be used across different 
communities. The IEEE is accepted as a neutral standards organization that could be successful at 
aligning and harmonizing relevant specifications; however, the project is in its infancy and no standard is 
available for evaluation. 


3.5.6 AETC 


In addition to participating with and monitoring the various public-private partnerships, ADL Initiative 
staff collaborated with AETC’s Airman Learner Record (ALR) program office to understand how this 
record might be used across the DoD education and training enterprise. The ALR captures and 
consolidates the entirety of an airman’s training, education, experiential development, and 
competencies obtained on the job, off-duty throughout their career, and prior to joining the military. 
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In 2019, the ALR was focused on assessing functionality, developing and evaluating the front-end design, 
and validating requirements for three major use cases: Learner, Civilian, and Leadership. In September 
2019, the ALR 1.0 framework was delivered to the Air Force Learning Services Ecosystem (AFLSE) test 
environment. The ADL Initiative continues to meet regularly with the ALR program office to monitor 
progress, ensure technical compatibility with other TLA standards, and better understand Air Force 
requirements as they continue to evolve. 


3.5.7  ELRR 


The DoD’s Enterprise Digital Learning Modernization initiative, through the DoD CMO, has also issued 
guidance to develop a common learner record repository. The Enterprise Learner Record Repository 
(ELRR) intends to track an individual’s summative experience in education and training across systems, 
platforms, and organizational boundaries in the form of competencies and credentials. Data standards 
are required to support federated data contracts and services that can integrate data from authoritative 
sources owned by DoD organizations (e.g., Army ATIS, AETC ALR, Navy Electronic Training Jacket). These 
connections will be managed by Authorities to Connect. 


The ELRR stores the digital version of each credential (or micro credential) and assertion of competence, 
with pointers to the evidentiary chain of local records from each location maintaining records of the 
learner’s career history. Assertions include other information such as physical/psychological/behavioral 
attributes, personal preferences, and competencies not associated with a credential (such as second 
languages). Each data element is properly tagged to implement privacy settings in conjunction with 
federated identity, credential, and access management. 


As the requirements for credentials change over time, there is no architecture in place to track the 
impact of those changes. A true ELRR capability will rely on expanded use-cases that look at both civilian 
and military requirements across different time scales, from in situ learning to an entire career arc. For 
example, learner data such as engagement, preferences, misperceptions, misconceptions, and 
motivational attributes can provide a wholistic view of the learner, which can optimize learning at the 
activity and course level to increase the velocity toward the learner’s goal. 


3.5.8 Control Loops 


Lifelong learning data that are stored within learner records can be abstracted into five control loops 
that describe the continuum of learning throughout a career, as shown in Figure 11. These control loops 
provide a framing mechanism for the TLA data strategy where each control loop has its own data 
sources and sampling requirements. Optimizing the results within each control loop, through data- 
driven decisions and eventually automation directly supports mission effectiveness. 


e Control Loop One (Learning Activities) involves the data required to optimize the current learning 
activity. Control Loop One optimizes performance in the task at hand. 


e Control Loop Two (Credentials) involves the data required to support the planning of learning 
activities in pursuit of credentials, which may require completion of multiple courses. 


e Control Loop Three (Job) involves the data required to optimize competencies and credentials in 
support of an individual’s current job. This includes feedback mechanisms for de-credentialing if 
proficiency is not maintained in certain skills. 
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e Control Loop Four (Career Arc) involves data to support the planning, placement, and evaluation of 
individual career growth, including planning development trajectories that are aligned with 
organizational needs. 


e Control Loop Five (Career Transition) involves selecting a new career option, such as selecting a new 
Occupational Specialty or pursuing a substantively different line of work. 


Mission 
Effectiveness 


Formal | Informal Work/Life 
Training Training _ Experience 


Credential 


Job/Duty/Gig 


Career / MOS / AFSC / NEC 


Control Loop 5: New Career Goals 


Figure 11. TLA Control Loops. To reduce complexity, the TLA decomposes lifelong learning into a series of five 
control loops. Each loop focuses on a different aspect of individuals’ learning paths throughout their careers. 


There are numerous challenges in creating a lifelong learning record that still need to be determined. 
The IEEE 1484.2 ILR and the IMS Global CLR are both promising solutions but there are more questions 
that will need attention in 2020. What elements of the learner profile should the learner be in control 
of? How does learner information pass from one organization to another? What Authoritative Systems 
have permissions to write to the learner’s profile? How do we represent learner context? 


Taken together, these different learner record initiatives set a fast drumbeat toward establishing a 
knowledge base allowing individuals and employers to gain deeper insights into required workforce 
competencies and learning achievements. 


4.0 2019 TLA REFERENCE IMPLEMENTATION 


The 2019 TLA Reference Implementation provided the test and evaluation environment for conducting 
research required to validate TLA specifications, standards, technical designs, and other TLA-related 
services. It also established a 24/7 testing capability to evaluate different approaches for federating 
data. 


The 2019 Implementation built upon lessons learned from 2018 to better represent a true ecosystem of 
microservice-based enclaves of data, services, and learning devices federated across multiple locations. 
A key focus for 2019 was to replace the point-to-point communications of the 2018 system with a 
modern publish/subscribe (pub/sub) type message service. Pub/sub services are much more scalable 
than point-to-point links and are more representative of the architectures used across the DoD. This 
allowed the ADL Initiative to better evaluate different TLA components in the context of an operational 
environment. 
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Another 2019 focus was to decouple the learning management functions of an LMS into composable 
“microservices” that communicate using scalable and open protocols. The TLA engineering team chose 
Kafka, a streaming type messaging service, to provide the pub/sub infrastructure. The research 
uncovered the impact of using streaming messages on the system topology, and the impact on 
standards such as xAPI, defining the structure of the passed messages. The team evaluated the 
proposed core/edge architectural pattern and the performance of the Kafka data streaming service. 


4.1 Design 


The Reference Implementation is comprised of three core microservice groups, a data lake composed of 
the four data stores associated with the TLA data strategy, and the Kafka pub/sub messaging service. 
This approach was used throughout 2019 to conduct numerous experiments for testing the architectural 
designs, integration with streaming messages, and the types of data and protocols used at scale. 


The TLA topology relies on the heuristic of distinguishing core services from edge systems. Core refers to 
mandatory functions and data sources, while edge refers to optional or externally federated systems or 
data sources. The System/Sub-System Design Document attached as Appendix D describes the detailed 
logical, hardware and software architecture, components, interfaces, and concept-of-execution for the 
Reference Implementation. Hosting is provided through Amazon Web Services (AWS), coordinated by 
USALearning with an Ubuntu 16.x, Hadoop, Docker, and VMWare component stack. 


While 2019 focused on the establishment of a development cluster, the Reference Implementation is 
designed to support a three-tiered configuration management pipeline that includes a development 
environment, a testing and evaluation environment, and production environments. Additionally, the 
development operations (DevOps) pipeline made it easier to compose the final system by progressively 
adding components or microservices deployed as Docker containers. 


4.2 Requirements 


Using concept maps and design structure matrices, basic LMS functionality was decomposed into 
functions, attributes, and data. This analysis provides the foundation for identifying the physical 
deployment of data and services, which are collected into three core and two back-end service groups. 
The behavior and functionality of each service is defined and aligned with TLA business functions. Input- 
output data flows are identified and aligned with the required TLA data stores. TLA Functional 
Requirements are attached as Appendix A. 


It is impractical to have a single repository for each type of data for all DoD components because of 
horizontal scale, multi-level security, and implementation costs. Thus, learning technologies, such as the 
LMS were abstracted away to edge systems. As shown in Figure 12, the services layer acts as the bridge 
between learning devices, other TLA components, and federated data stores across the DoD enterprise. 
The federation requires standardized interfaces and governance to maintain identity management and 
integrity of authoritative data sources between installations. 


The services and data communicate with the web-standard Representational State Transfer (REST) 
pattern based on the Open API and HTTP. Each service exposes stored data to an application so they can 
be transformed into meaningful information used by other TLA components. Each service includes 
control logic and user interfaces for a set of functions. The data contracts between data and service 
layers are shown based on the nature of the data exchanged. 
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Figure 12. Core TLA Functions and Data. Core services provide access to the core data structures and act on data 
provided from learning technologies to manage current learner state. 


4.3 


Core TLA Software Components 


The TLA Reference Implementation encapsulates core technology platforms, microservices, and data 
equities that may be federated, combined, linked, and merged as the ecosystem grows. Thus, the term 
core does not mean centralized. The core services and data identified in Figure 12 include: 


Competency Management - This refers to a set of services that manage evidence of individual or 
team KSAOs. Within the TLA context, the xAPI activity streams stored in the LRS provide the 
evidence upon which competency assertions are made. Evidence is aligned with competency objects 
that are encapsulated into an RCD Competency Framework. These services also verify chains of 
evidence, handle the associated trust functions, generate authoritative assertions of capability 
(credentialing or de-credentialing), and makes estimations of competence based on granular or 
inferential data. 


Learning Event Management - This refers to a set of services concerned with scheduling learning 
activities and capturing data output from those activities. Scheduling learning activities may involve, 
for instance, a scheduling service, potentially aided by a recommender or content curation service. 
This component consumes xAPI statements from the Kafka stream and relays that information to 
the learner profile. It triggers relevant MOM events for the statement and updates paradata. 
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e Activity and Resource Management - These services are concerned with the registration and 
maintenance of accurate references to the location of learning resources and their metadata, as 
well as the verification of the resources required to access them. Key features include the ability to 
generate and manage metadata, taxonomies and ontologies, the alignment of content with 
competencies, paradata, semantic search services, and machine-actionable metadata. 


e Learner Data — The Learner Profile - Each TLA enclave is expected to house local learner records, 
with user data and locally generated assertions of competence. User data includes learner state and 
personal attributes relevant for local learning and reporting. Local learner records may also link to 
external learner records (e.g., global preferences collected from the DoD ELRR) that support local 
actions, such as learning event adaptation and scheduling. The competency or credential evidence 
earned locally, stored initially in a local learner record, can be forwarded to these other systems, 
and conversely, federated competency or credential data from edge systems can be ingested into 
the local learner profiles. 


e Competency Data — The Competency Framework - Competency Frameworks store the data 
describing competencies, their granular components (e.g., KSAOs and information like attitudes and 
motivation), and relationships among individual competencies (e.g., prerequisites and co-requisites). 
Also, these frameworks relate competency data with measures of performance and effectiveness, 
based on demonstration requirements for a given level of mastery in performance of a job or duty. 


e Learning Experience Data — The LRS - LRSs are the server-side component of the xAPI specification. 
They archive xAPl-encoded performance data from learning activities or experiences, which provide 
evidence of competence to the Competency Management Services. 


The mandatory LRS is known as a Transactional LRS which stores “actionable information” regarding 
human performance data. At the simplest level, the Transactional LRS can be analogous to a 
gradebook. The Transactional LRS contrasts with “noisy” LRSs, which may optionally be part of a 
given activity provider (e.g., a simulator or LMS) and are used to capture finer-grained information 
(e.g., each page turn or button press) for local analysis. This separation moves raw performance 
adjudication away from the core systems. 


e Data About Learning Activities — The Experience Index - An Experience Index stores local 
information about available Activity Providers and their available learning content. An Experience 
Index also stores information on the relationships among content, competencies, activity and 
content metadata, and evaluated paradata. 


Together, these data describe the learning experiences represented within the activity-content- 
competency triplet. The metadata are used for activity scheduling and evaluating the impact of a 
given learning experience on competence. Each TLA enclave may have its own Experience Index, and 
globally available elements listed in these Experience Indices can be federated into the ECC. 


Figure 13 shows how the 2019 Reference Implementation collected xAPI data from activity providers 
conforming to the TLA MOM. xAPI messages flowed through the other core systems, especially the 
Competency Manager, which generated assertions of competence based on the xAPI evidence. Learning 
Activities streamed statements to a local LRS. Each activity also streamed xAPI statements conformant 
to the TLA MOM to a transactional LRS. The Transactional LRS fed the Competency Management System 
to process MOM statements and update the Authoritative LRS. All xAPI statements eventually ended up 
in one of the two LRSs. 
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Figure 13. TLA Software Component Architecture. This diagram illustrates how xAPI data are used across the TLA 
to support different purposes. 
4.4 Back-End Services 


The TLA Reference Implementation evaluated different approaches to manage back-end services and to 
implement a DevOps Security pipeline to expedite integration and testing of all TLA components. The 
system of systems comprising the Reference Implementation were configured in accordance with the 
DoD CIO’s draft policy for FICAM. This configuration will be used to support additional testing to ensure 
the protection of PIl associated with learner data that is stored across the DoD enterprise as it is 
communicated to other TLA components 


e Identity Management - Personnel’s education and training data are currently locked in IT silos. 
Federated Identity Management allows organizations to securely and ethically verify learner data 
(e.g., evidentiary chain) across network enclaves and institutional boundaries through Authorities to 
Connect. 

e =©Virtualization Management - Generally provided by cloud solution providers, virtualization includes 
the services required to dynamically process addresses for virtualized networks, server instances, 
and data clusters (e.g., VSphere and Hadoop). Virtualization is closely tied to device management 
and cybersecurity. 


4.5 TLA Edge Systems 


TLA edge systems include LMS solutions, learning applications/devices, activity providers, web portals, 
back-end services, and externally federated data sources. Applications and Devices are xAPI Learner 
Record Providers (LRPs). The LRP is responsible for the xAPI compliant formatting and population of a 
Learning Record. Edge systems may have “boundary” services that normalize reporting to the TTA MOM 
profile and generate their own xAPI statements. 
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These normalized Learning Records can then be shared with the Transactional LRS. As the learning 
ecosystem evolves over time, edge systems may expand to include new types of instrumentation (the 
“Internet of Learning Things”) or additional data systems. 


4.6 Documenting the Objective Architecture 


Throughout the development of the TLA, the ADL Initiative’s development team used DoDAF views (see 
Appendix C) to document a common understanding across the team. The DoDAF mitigates complexity 
for specific stakeholders through the creation of viewpoints that describe how requirements are 
implemented and their impacts or benefits to each stakeholder. 


The DoDAF views are iterative and reflect the current thinking of the TLA technical team. These views 
will evolve as feedback is received, designs are modified, and technical specifications are updated. They 
are presented for discussion and debate across the education and training community, and feedback will 
be actively solicited from all DoD stakeholders. 


The 2019 DoDAF views are informed by the ADL Initiative’s experience in supporting active stakeholder 
projects across the Army, Air Force, and DHA. Additional insights were collected through participation in 
technical working groups and collaboration with industry partners and DoD stakeholder organizations. 
The TLA DoDAF includes the following types of views: 


e The All Views (AV) provide the TLA executive summary and a dictionary of terms. 

e The Capability Views (CV) describe the value propositions of TLA to the DoD Force Education and 
Training (FE&T) enterprise. 

e The Operational Views (OV) describe stakeholder roles, responsibilities, equities, and governance. 

e The Standards Views (StdV) show the specifications and standards from industry and government 
that define the TLA objective state. 

e The Services Views (SvcV) describe the topology and functions of TLA “nodes,” including core 
services, data stores, interfaces to learning devices, ancillary data, and back-end services. 

e The Data and Information Views (DIV) describe the TLA ontologies for governance processes, logical 
data descriptions, and schema definitions for the elements within the TLA. 

e The Project Views (PV) depict the programmatic aspects of the TLA research portfolio lines of effort. 


Each DoDAF view presents a subset of elements from the same underlying model of the architecture. 
The elements at one level support the elements at the next level. They document the objective end- 
state and projected value proposition for the TLA effort. 


4.7 Hardware Components 


The 2019 TLA Reference Implementation uses five virtual machines hosted on AWS, which provide the 
back-end platform hosting, virtualization, and Domain Name Service resolution functions. Each machine 
was procured under contract to USALearning and maintained by the ADL Initiative. 


The server instances communicate between themselves using either HTTP/S over TCP/IP or by 
producing and consuming messages to the centralized Kafka cluster, internally to the AWS campus. 
External clients accessing the portal, the hosted content, or the service redirects may be located outside 
the AWS campus and connect via REST, as shown in Figure 14. 
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Figure 14. TLA Physical Architecture. This figure outlines the high-level flow of information between the 2019 
Reference Implementation components. 


The physical and virtual servers communicate over a Virtual Private Cloud (VPC) network. The 
demarcation point provides network address translation between external clients and the VPC 
Reference Implementation resources (e.g., LMS web clients located housed by the ADL Initiative). The 
network itself is also virtual, with resources allocated across multiple subnets by the AWS services stack. 
The virtualization management was handled by Amazon back-end services. 


The TLA sandbox where the Reference Implementation is hosted is also home to different learning 
activity providers (e.g., Moodle LMS, Video Server, eBook player, and other mobile learning activities). 
Additionally, the NILE platform is hosted as an open source version of a commercial platform (NILE 
provides an individualized recommender service to track student competency, based on the common 
core model). NILE provided both a “federated LRS” and acted as a second “TLA enclave” for testing the 
Reference Implementation. 


4.8 Interfaces 


Communications between 2019 Reference Implementation components involved either HTTPS, 
WebSocket, or Kafka streams. Given the popularity and general simplicity of RESTful APIs in modern web 
services, HTTPS continues to facilitate most traffic within the 2019 Implementation with two exceptions: 
notification of new xAPI statements and learner profile updates. The primary protocol and data types 
are shown in Table 2 with fields listed as JSON being small datatypes that do not belong to an 
established specification. 
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Table 2. TLA Reference Implementation Protocol and Interface Matrix. Primary protocol and interface usage for 
each component within the 2019 Reference Implementation. 


Receives 
Al AR LP IdM CF CM LRS LEM Portal LA 


Serves 


Experience (XI) 
Activity Index (Al) 
Activity Registry (AR) 


JSON JSON 


Learner Profile (LP) 


Identity Manager (IdM) 


Competency Framework (CF) CASS 


Competency Manager (CM) 


Transactional LRS (LRS) 


Learning Event Manager (LEM) 
Portal 


Learning Activities (LA) 


Key: HTTPS | Kafka 


These services and data structures were reviewed with ADL stakeholders to evaluate their applicability 
to all DoD components. The minimum materiel requirement for the 2019 Reference Implementation 
included the Transactional LRS, the Kafka messaging service, the Experience Index, the Identity Manager 
(i.e. Keycloak) and the create/read/update/delete (CRUD) service. Full integration of the Competency 
Management service is delayed until a CaSS hardening project is completed in 2020. This work is being 
done to support an /nterim Authority to Test for evaluating CaSS and to measure return-on-investment 
for competency-based learning. This work will initially involve competencies developed to support Air 
Force Specialty Codes but is expected to expand to support additional experimentation with the Army 
and Navy. The TLA testbed currently includes a static message test harness that emulates CaSS 
messages. The Learning Event Manager is still under development. 


4.9 TLA Sandbox 


The 2019 TLA Reference Implementation was configured to support additional testing of stakeholder 
modernization efforts related to education and training. Other ADL Initiative projects are currently 
hosted within the TLA sandbox and will be integrated into the overall testing and evaluation capabilities 
available to ADL stakeholders. The DATASIM project is creating an authorable xAPI simulator that ingests 
xAPI Profiles and generates learner data at the scale and scope required to support DoD. The Data 
Analytics and Visualization Environment (DAVE) is providing an open source analytics toolkit with a 
library of data cards, analytics algorithms, and visualizations that can be tailored to provide different 
types of analytics depending on stakeholder needs. 


5.0 TLA TESTING AND EVALUATION 


Architecture development is governed by heuristics”*. The general heuristic for the TLA service topology 
was that the goals of the future learning ecosystem mirrored many “business to business” (B2B) web- 
based solutions that moved from single companies with their own software to integrated supply chains 
using interoperable business software. A common pattern in B2B processes is the Enterprise Service 
Bus, which separates the diverse set of data sources as “edge” systems, linked by centralized “core” 
data services that handle the translation and transport functions between them. 


24 M.Maier & E. Rechtin. (2002). The Art of Systems Architecting. CRC Press, Washington D.C. 
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The primary governing heuristic for the architectural refactoring effort was to decouple functions from 
data stores, decouple system performance from high uncertainty elements, decouple system 
performance from critical nodes, and decouple components from proprietary interfaces. The method for 
defining the details of the learning service and data topology was a data-driven analysis followed with a 
design structure matrix”®. This matrix is a variation of N? chart analysis, based on the linear decoupling 
principle. A secondary heuristic governing the service topology was managing computational complexity 
to ensure that the proposed architecture would retain acceptable performance levels at the scope and 
scale required to support an enterprise DoD architecture. 


This section describes the analytic methods used to develop each design hypothesis and organizes them 
within the experimental plan. Based on an analysis of a 2018 TLA Test and Demonstration, five high-level 
goals were adopted for deploying the 2019 Reference Implementation: 


e Implementation of communications at scale - Demonstrating the use of Kafka streaming to 
implement a pub/sub messaging service instead of point-to-point, as pub/sub is more scalable and 
more resilient to morphology changes. 


e Decoupling of components — Demonstrating the removal of dependencies on a central state 
manager by establishing a core/edge microservices based architecture. The 2018 Reference 
Implementation relied upon a recommender system that acted as a single state manager across all 
data components. 


e Federation of data — Evaluating options to enable horizontal scalability and preserve local command 
equity in data. The TLA architecture is composed of different network enclaves, communicating in 
federations, within the ecosystem that complies with the specifications of the ecology. 


e Evaluation of candidate standards — Integrating stakeholder programs of instruction into the TLA 
Reference Implementation and migrating to appropriate TLA standards to evaluate their ability to 
meet stakeholder needs. 


e =Risk buy-down for DoD modernization — Evaluating technical approaches, commercial solutions, and 
related policies in support of the DoD CMO Enterprise Digital Learning Modernization effort. 


The data federation experiments included an independent variable with multiple combinations of data 
nodes and users, as well as multiple network configurations for the test federation. The dependent 
variables included the observations regarding throughput performance and feasibility. Results were fed 
back into the requirements, the underlying architecture, and the specifications documents that 
ultimately comprise the TLA. This work was also incorporated back into the Reference Implementation. 


5.1 Enabling Communications at Scale 


Message streaming services like Kafka are high quality-of-service, high throughput technologies used to 
support the development of scalable web-based solutions. They are similar to other pub/sub services 
where components react asynchronously when necessary but aren’t required to react to every message 
or transmit full changes in each message according to a predefined schedule. However, data streams 
must be unaltered as they are communicated from the service or edge system creating the message to 
the TLA data lake. This demands consideration of the data topology for systems because any data 
transform, or modification of the message payload, must be transmitted as a new data stream. 


25 Nam Suh. (2001). Axiomatic Design: Advances and Applications. Oxford University, England. 
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Because any data transform gets transmitted as a new data stream, there is potential to create logic 
loops when communicating these data back to the originating system. This is especially hazardous in 
systems that use linked data, as these data must exist in complete form before and after the statements 
referencing them are generated. 


The experiments for vetting this design relied on the Agile software development technique of 
continuous integration and test. The TLA technical team verified the core and edge design paradigm 
through peer-review design meetings held with various standards organizations, working groups, and 
industry associations. The general concept is that learning activities generate messages via xAPI 
statements streamed through a Kafka broker. Core TLA systems subscribe to this data stream and 
generate additional data streams that define and refine the context of learner performance within each 
activity. As additional data streams are generated by core components, they also stream through the 
Kafka broker and are categorized based on the TLA data strategy. 


Key to maintaining the event-driven streaming topology was the establishment of the TLA MOM. This 
results in an event-driven design where events described in the TLA MOM propagate changes in a 
learner state. The data for verifying the MOM came from case studies built around stakeholder efforts. 
Each of the TLA’s core systems was configured to listen to all xAPI statements to test the viability of a 
learner-centric data model. Five sets of 200 users were simulated for different learning activities. These 
were emulated by sending TLA MOM-compliant xAPI statements to the TLA’s Transactional LRS in an 
order that follows the learning lifecycle. The Transactional LRS forwarded all statements to their 
respective Kafka data streams to trigger an update to the learner’s state. 


This highlights the complexity of the data topology within any streaming architecture. It also becomes a 
caution when using linked data, as the linkage must preserve the ordering of statements. Statements 
that link to another concept must do so after the first concept is instantiated with a statement or they 
will cause a memory fault, corruption, or crash. In this configuration, there are streams that “loop” 
around to create those references. 


The Kafka system can manage millions of xAPI statements, which presents some unique challenges. The 
xAPI specification recognizes an LRP as a system that generates an xAPI statement, and a Learner Record 
Consumer (LRC) as a system that uses xAPI statements. In the Reference Implementation, TLA core 
components accomplish both these capabilities. If chained together in a serial fashion, data may be 
obfuscated, and the total throughput of the system may be diminished as the same messages are 
created and transmitted repeatedly. A reverse proxy was used to establish an LRS-agnostic mechanism 
for forwarding xAPI statements, resulting in a parallel bus of LRPs, including not only edge systems but 
also core systems generating MOM verbs, as shown in Figure 15. 


Learning Record Learning Record Learning Record Learning Record 
Provider Provider Provider Provider 


Learning Record Learning Record Learning Record Learning Record 
Consumer Consumer Consumer Consumer 


Figure 15. Parallel Bus Topology of Learning Record Providers and Learning Record Consumers. 
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5.2 Decoupling TLA Components 


The event-driven streaming topology for services and interfaces was key to decoupling TLA functions 
and data. This allowed independence between each component internally and allows the alignment of 
TLA components to other stakeholder modernization efforts. Each service reacts to events, without 
having to know the sequence of events and services upstream or downstream. All events eventually end 
up as persistent records in the data lake. 


The 2019 design had to enable a true ecosystem, where different learning theories and approaches can 
be used with a dynamic mix of learning activities. The use of the services approach discussed in the TLA 
DoDAF dictates the presence of learner-centric event messages that trigger services to perform data 
functions. Each of these functions could happen in isolation, independent of their order in the overall 
execution, meaning that the overall system is not “stateful.” In the 2019 Reference Implementation this 
was accomplished by normalizing everything to the TLA MOM profile. 


The TLA MOM provides guideposts for filtering data to support the presentation of information in the 
correct time horizon for the appropriate TLA control loop. The MOM maintains a focus on collecting 
actionable information from learning activities and forces the adjudication of performance to occur at 
the edge. The learner is the only component of a learning solution guaranteed to be present, so the 
MOM profile normalizes data about learner actions and the context of how that learning takes place. 


The experimental design for validating the MOM includes the standup of multiple federations, 
demonstrating a range of potential configurations, internal components, and associated edge learning 
activities. These include legacy LMSs, adaptive learning systems, simulators, or other learning devices. If 
the MOM can be used with minimal changes to the specification and minimal changes to systems being 
integrated, it will provide evidence of architectural stability. The following questions were used to 
evaluate the stability of the MOM qualitatively: 


Does the MOM accommodate all required /earner state data? 

Is the learner-centric model a viable technique? 

Is streaming all xAPI data a viable data management technique? 

Is using xAPI messages to create a ledger of learner state a scalable approach? 

Will the xAPI data generate an auditable chain of evidence? 

Do the xAPI data generated give insight into adapting the learning environment to the learner? 
Does the MOM accommodate a range of learning activities and theories, including LMS, intelligent 
tutors, and self-regulated learning devices? 

8. Does the MOM capture enough context information about the learning environment to allow 
analyses of learner preferences? 


SOI Ba rS 


Part of this work includes contextualizing the evidentiary chain of xAPI verbs included within the MOM. 
Each xAPI statement represents trusted xAPI data that determines the learner’s location within their 
career trajectory. All trusted statements include a link to the statements used as evidence, as well as to 
the nodes within the Competency Framework they reference. In the current reference implementation, 
the creation of conferrals and assertions is simplified for testing purposes to monitor response time and 
data loss. In an operational TLA system, the creation of authoritative statements would be logic driven, 
including layers of business logic encoded in the Competency Framework. 
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The decoupling experiments generated evidence to validate the TLA MOM event trace in place of a 
singular state manager by sharing data across network enclaves. As shown in Table 3, this also allowed 
the team to verify the TLA standards used to support the TLA data strategy and allowed benchmarking 
for performance. Performance was evaluated based on the number of users, number of records, and 
across different network topologies to emulate the scope and scale of data generated across the DoD. 
Federated Identity Management was tested using Open ID Connect and benchmark testing for search 


and retrieval times. 


Table 3. Subordinate Technical Objectives for Establishing a Decoupled Architecture. The results of these technical 
outcomes are captured in the DoD Architectural Framework (DoDAF) views and design artifacts. 


Element 
Define topology 
for components 
and interfaces 


Purpose 
Decoupling of data and services, in 
order to better provide an adoption 
and migration strategy. 


Comments 
Migration defined in the TLA Capability Maturity 
Model (CV-3) and decoupling of core and edge 
services and data defined in SV-1. 


Minimize stateful 
nature of system 


Part of enabling a true “ecosystem” 
which increases resiliency. 


Enabled through the TLA MOM and the DoDAF 
which allows for event-driven services. 


Enhance 
composability of 
ecosystems 


Accommodate ad hoc mix of learning 
technologies without re-engineering 
efforts for every change. 


Defined the TLA enclave as a function of 
interfaces and governance structures (OV-2). 


Enhance 
performance at 
scale 


Verify use cases to ensure that 
solutions based on proposed standards 
and best practices will scale. 


Conducted experiments into federated identity 
and federated data (LRS, Experience Index, etc.). 


5.3 


Federation of Data 


The 2019 Reference Implementation provided an operational context for the development of key 
enterprise level technologies required to field the future learning ecosystem. This line of research was 
informed through coordination with external stakeholders, OUSD(I), and the Army Simulation and 
Training Technology Center (STTC), specifically focusing on issues required to: 


e Field an accredited LRS to move toward Capability Maturity Model (CMM) Level 1; 
e Establish an ECC; 
e Establish Federated Identity Management to support enterprise analytics with the LRS data; and 
e Enable the ELRR. 


This experiment hypothesized that the ECC might be deployable as a service searching from multiple TLA 
compliant enclaves’ Experience Indices, rather than making a copy and managing from a central 
repository. Federating a Learning Experience Index to a central service allows learners to discover new 
content and experiences as updates are published. For the TLA to resolve learning experiences between 
enclaves, administrators must register the local Learning Experience Index with a global Experience 
Index. For this experiment, it was assumed that learning experiences are already unique within the TLA 
federation, as data equity and governance processes will manage the transition to a common catalog 
and maintain configuration over its life. 


As shown in Figure 16, the 2019 research tested the scalability performance of a federated Experience 
Index. A challenge arose when multiple TLA enclaves owned similar courses that map to a similar 
competency. These similarities or differences needed to be aligned during the learning activity 
registration process. This is a process where metadata about a learning activity was uploaded to the 
Experience Index so that all TLA components were aware of its existence. This research employed an 
augmented version of LRMI as a transmittal format for exchanging data that federated to the ECC. 
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Figure 16. Federated Experience Indices Throughput Performance. Benchmark tests of 50 sites with between 5- 
400 records for a total of 10,000 courses, representative of the data size and loading for a distributed ECC. 


A containerized activity registry and resource management service with its own Experience Index was 
put onto an AWS server instance. A local hosted Experience Index was also configured. A script was 
written to load example metadata to both learning Experience Indices. The main function of the local 
Experience Index was to lookup activities and content within TLA enclaves. It was load tested over a 
Wide Area Network with 10, 100, 500, and 1000 simultaneous users all making queries to the federation 
service for 25 minutes to simulate realistic loads for this service at any given time. 


The LRMI metadata schema allows anyone who publishes or curates educational content to provide 
education-specific metadata about their resources with confidence that major search engines would 
recognize it. As part of a linked-data strategy, metadata and Competency Framework information must 
trace back to a single semantic authority through a schema server. 


5.4 Federated Identity Management 


In the management of federated learner identities, one challenge was that not all systems store user 
information the same way. Learners may also use different local account names for different purposes. 
This is especially relevant to the Intelligence Community, where individuals maintain different personas 
on different networks. There is no guarantee these accounts, or systems know about each other, so a 
federated approach to Identity Management is required to resolve this issue. 


For a TLA compliant federation of multiple enclaves and/or devices to resolve multiple accounts, the 
learner must register an account with the TLA. To do this, the learner logs into the TLA and selects an 
external ID provider. The learner then logs into the external provider’s account and registers that 
account as an alias to their master account within the TLA. 


This system minimized human interaction, and no Pll was transmitted throughout these transactions. 
However, the inclusion of Pll is still a possibility as an LRP could choose to use PIl as its actor name. The 
TLA federation negotiation rules and device registry will address this by requiring non-human 
identifiable actor IDs and synchronizing between device and core systems. 


To test the feasibility and scalability of this approach, a containerized learner ID management service as 
described previously was put onto an AWS server instance with an 8GB hard drive. A script was written 
to register 100, 1K, 10K, 20K, and 100K users, each with three different aliases. After an initial slow 
response rate, the database of learner aliases was indexed and the map retrieval was nearly consistent, 
as shown in Figure 17. 
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Figure 17. Indexed Results. The Reference Implementation uses a containerized learner ID management service for 
test and evaluation. Proper indexing substantially improved performance. 


The evaluation of candidate standards and risk buy-down goals for testing and evaluating the TLA are 
outlined in previous sections. Section 3 Research Into Commercial Standards outlines the work 
performed to evaluate candidate standards while this section reviews the architectural considerations 
for the DoD CMO’s Enterprise Digital Learning Modernization initiative and other DoD modernization 
efforts. The results of these experiments were used to inform functional requirements and next steps in 
the development of the ECC and ELRR programs. 


6.0 CONCLUSION AND NEXT STEPS 


The principle technical objectives of the 2019 effort included evaluating candidate standards, replacing 
the point-to-point communications, decoupling system components, and exploring federated data 
strategies. These objectives were successfully met during the 2019 work on the TLA. 


6.1 Research Results and Successes 


The 2019 Reference Implementation migrated to a core/edge microservices based system using Kafka 
streams as a Communications method. The switch to Kafka required iteration with the TLA MOM to 
identify the correct activity streams and ensure an event-driven system with the correct flow of 
messages between components. The streaming capability allowed the team to capture performance 
benchmarks using different technical approaches for key TLA components. The design effort yielded 
several tangible results for validating the required TLA standards, as well as a wealth of best practices 
and lessons learned. 


As the granularity of the TLA’s scope increased, naming conventions became a hurdle. TLA component 
name choices and the subsequent responsibilities of services sharing those terms caused duress 
throughout the design and development process. Activities, content, resources, and experiences are 
tightly coupled concepts with distinct places in the 2019 Reference Implementation and the TLA overall, 
but these were synonymous for 2018. As such, services carried over from 2018 became tedious to 
maintain and adapt to 2019 updates due to their now-contradictory naming conventions. 
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A basic feature of any system is its cybersecurity posture, including privacy, non-repudiation, integrity, 
security, and reliability. Technical personnel must coordinate closely with these policy owners for key 
capabilities including, Memoranda of Agreement and Authorities to Connect for federated systems, 
authorities to operate in multi-level security cross-domain environments, network and server 
virtualization, dynamic endpoint management, and Global Information Grid integration (see the DoDI 
8500 series). 


Similarly, the federated data structures will require broad coordination for networking and file 
management services (e.g., federated IDs), maintaining data integrity and searchability (e.g., the ECC), 
and asynchronous data management (e.g., for deployed units or individuals). Finally, the identity 
management requirements within the DoD will necessitate ongoing coordination for virtualized identity 
management, encryption, tokens, digital signatures, universally unique identifiers (UUIDs), and PII 
protection, with the latter including clean-room server policies. 


The overarching goal of the TLA research portfolio is the selection or generation of the standards, 
specifications and policies necessary to instantiate the future learning ecosystem. This includes the 
interconnected learning devices, computational assets and data equities, as well as the enterprise level 
approach for learner records, an ECC, federated identity management, and enterprise analytics. 


6.2 Policy Guidance 


The DoD Instruction 1322.26”° establishes policy, prescribes procedures, and establishes information 
requirements for developing and deploying distributed-learning content for DoD military and civilian 
personnel. Fungible references address emerging learning technology concepts and challenges by 
providing additional policy guidance or implementation instructions related to education and training. 


While the ADL Initiative is the steward of the DoDI 1322.26, the fungible reference updates are matured, 
validated, and approved by the Defense ADL Advisory Council (DADLAC), composed of military and 
civilian leaders in the distributed learning field. As the TLA matures, incremental updates to the fungible 
references will include guidance to help DoD stakeholders confirm TLA standards are implemented to 
ensure the interoperability of learner data. 


The future learning ecosystem described in this report requires the selection or creation of standards 
and specifications across two categories: data metamodels to ensure semantic interoperability 
(presentation layer), and data interface models to ensure connectivity (transport/application layer). 
Interface standards are built from web commerce protocols. Interservice communication uses REST 
protocols. Event traces among components are normalized as activity streams using the MOM xAPI 
profile. The launch of remote devices uses Learning Tools Interoperability?’ (LTI) protocols. Remote ID 
authentication uses Open ID Connect”® (OICD) protocols. 


The data metamodels describe the data within the four pillars associated with the TLA data strategy: 


e EEE P9274.1 xAPI 2.0 — Learning activity tracking uses the xAPI and JSON standards to capture 
learning activity streams. The cmi5 specification and the TLA MOM are also contained within this 
data pillar since they contain an xAPI profile. xAPI 2.0 is targeted for approval in 2020. 


26 DoD Instruction on Distributed Learning - https://www.ad!Inet.gov/policy-dodi 
27 IMS Global LT! - https://www.imsglobal.org/activity/learning-tools-interoperability 
28 OpenID Connect - https://auth0.com/docs/protocols/oidc 
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e EEE 1484,12.1 LOM 2.0 — Descriptions of learning activities and their associated content are stored 
in the TLA’s Experience Index and use a modified version of the LRMI standard. 


e [EEE 1484.20.1 RCD - The Competency Framework includes the definition of a competency, the 
relationship to other competencies, and the alignment of evidence to help measure proficiency of 
the competency, are included in the RCD standard. This standard is expected for approval in 2020. 


e EEE 1484.2 ILR or IMS Global CLR - The Learner Profile will include credential sharing with 
OpenBadge 3.0 and the CTDL. 


Implementation guidance and business rules for how to use these interoperability standards is 
established by IEEE 1484.1 CMALTS, a broad standard that is equivalent to the overarching TLA 
specification. Device federation will likely be a NIST standard as it is driven by device management 
cybersecurity concerns. Data access and device registry for anything located outside of the enclave 
firewall must be consistent with cybersecurity policy, to maintain security, integrity, system availability, 
and non-repudiation of enterprise data. 
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Figure 18. The TLA Ecology. The TLA defines metamodels, interface specifications, and business rules that define 
requirements for making existing learning technologies interoperable at the enterprise level. Organizational 
governance procedures allow for progressive adoption, migration, and growth. 


Figure 18 depicts the relationship between DoD data equities, governance, policies, and TLA 
specifications. Learning system features need to be separated into services associated with the data 
structures they support, and they must be deployed within and across TLA compliant enclaves. Device 
and data connections are governed by DoD cybersecurity policies. The specific component design may 
differ, as long as the business rules, data metamodels, and exchange protocols are consistent. 
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6.3 Migrating DoD Systems to the TLA 


The migration of existing DoD education and training systems to the TLA will be phased over time and 
can be modeled as a CMM, shown in Figure 19. Sites migrating toward TLA capability maturity start at 
Level 1 by migrating from a SCORM managed, LMS-centric solution to an xAPI conformant solution 
where learner performance is stored within a centralized LRS. The LRS provides analytics beyond the 
“counting and completion” statistics typically available within an LMS. 


Level 2 supports the aggregation and analysis of learning data at the enterprise level. It requires 
enterprise level services to track global progress and the adoption of governance procedures to 
standardize learner identity, data labeling, and data reporting to enable enterprise analytics. Level 3 
adoption further decouples learning solution functionality within the enterprise. Each function 
previously performed by a single LMS is now a decoupled service or set of services. Federation is driven 
by cybersecurity policies and is enabled by enterprise-level services developed in Level 2. 


CMM Levels 4 and 5 focus on automation and full integration into the human capital supply chain. These 
levels represent the objective state of policy, technical specifications, and standards that will enable the 
future learning ecosystem characterized by interconnected devices and interoperable data. The 
standards that comprise the TLA technical specification act as a “distributed ledger” for learner data that 
is globally discoverable and usable across the DoD enterprise. This capability enables efficiencies for DoD 
stakeholders across the entire lifecycle of education and training including the identification of need, 
discovery of resources, and evaluation of learning opportunities to enable capable manpower. 


6.4 Governance Considerations 


Governance for a system as complex as the TLA will require steps at multiple levels within the overall 
system of systems. Global governance occurs at the DoD enterprise level and requires the maintenance 
of global metamodels for describing learners and competencies. A minimum set of metadata attributes 
for the ECC and ELRR initiative is also needed. Coordination, continual review, and regular updates are 
necessary for federated data and systems. Governance procedures must continuously address data 
structure compatibility, message interoperability, coordination between enclaves, and data mappings. 


Global governance will include new organizational processes and DoD-wide policies that facilitate the 
collection, analysis, and use of data across the DoD enterprise. Different DoD functional components will 
be responsible for governing different aspects of cybersecurity or interoperability applicable to their 
own command’s data equities. Local and regional governance will define ownership and labeling 
requirements. Agencies and military command organizations, for example, will have attributes they 
must gather and evaluate that are unique to their organizations. 


Other governance considerations include the need to federate the development and maintenance of 
Competency Frameworks across DoD functional components. Competency Frameworks will be 
comprised of numerous competency owners that maintain and update specific parts of an overall 
framework. This is already prevalent across the DoD and is exhibited by the Air Force Competency 
Directorate and the U.S. Army Training and Doctrine Command. 
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Figure 19. TLA Capability Maturity Model. The CMM allows for gradual migration of legacy systems to a 
microservice-based infrastructure of core services that federate data across DoD components. 
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6.5 Future Research and Next Steps 


The 2019 TLA Reference Implementation established a solid foundation for more robust 
experimentation in 2020. By migrating to a more modern data streaming approach, the Reference 
Implementation can be used by stakeholder organizations to test and evaluate the functionality and 
interoperability of their TLA-compliant systems. This capability will allow 2020 collaborations with the 
Defense Acquisition University, the Army’s STEEL and Squad Performance Model projects, AFLSE, DHA, 
and others. 


As the DoD migrates from legacy “data silos” to more integrated systems, it must start with the 
organization and technical structures from the legacy systems. Education and training records of DoD 
personnel are distributed across a variety of systems and locked into countless disparate data formats. 
Transport, control, management, governance, and ownership of such data are not easily 
accomplished—particularly across technological and organizational boundaries. The ADL Initiative’s 
2020 TLA research agenda adds new milestones to several 2019 projects, advancing toward the 
transition of foundational TLA capabilities. The 2020 research goals include: 


1. Federated Identity and Access Management. Data must be accurately aggregated across education, 
training, and operational learning experiences to create an accurate portrait of personnel and their 
learning states. For education and training systems there is no cross-domain solution for connecting 
across enclaves. The DoD requires a way to securely integrate learning data across these boundaries 
while maintaining an individual’s unique identity across systems and meeting cybersecurity and Pll 
requirements. Identity management includes not only UUIDs for personnel but also the internal 
references for managing learning resources. This work will investigate best practices for 
anonymizing actor fields in an xAPI statement so that xAPI can be transmitted in accordance with 
DoD policy. 


2. Enterprise Learner Record Repository. The ELRR must combine data from locally maintained learner 
profiles with globally coordinated learner data. These data must be (safely and ethically) portable 
and auditable across time, installation, and institutional boundaries. Specifically, the ELRR 
represents the entire career trajectory for each learner within the DoD enterprise. This line of effort 
requires participation in the multiple industry and standards initiatives working to harmonize 
learner record standards. It also requires the development of an Initial Operational Capability that 
can be used to support an Advanced Concepts Technology Demonstration (ACTD) that shows how 
these capabilities can create efficiencies across the DoD and begins to measure the potential return 
on investment. 


3. Enterprise Course Catalog. To build a learning ecosystem, instructional activities across the system- 
of-systems must be discoverable and accessible. This is achieved with the ECC, a service that indexes 
the available learning resources. The ECC is built from linked Experience Indices, with local listings of 
learning activities, their available content and resources, and their alignment to a Competency 
Framework. The 2020 work requires participation in creating the LOM 2.0 standard and in the 
development of an Initial Operational Capability that can be used to support an ACTD that shows 
how local course catalog listings can populate local experience indices that federate to the global 
DoD ECC. The ACTD approach intends to provide a mechanism to test and evaluate the ECC 
prototype while measuring the impact, benefits, and return on investment. Automated tools for 
creating metadata from local course listings will also be evaluated. 
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cmi5 Viewer and Conformance Test. The cmi5 specification is critical for recapitalizing SCORM 
content. The cmi5 specification defines how LMS solutions launch content using xAPI as the content- 
to-LMS communication layer. Many of the underlying standards used in SCORM do not sufficiently 
support the myriad technologies used in modern learning environments. The cmi5 specification was 
created to replicate the SCORM content package to replace SCORM in the delivery of online courses 
and traditional computer-based training. Research will focus on producing a freely available, open 
source cmi5 content player and conformance test. This work is expected to be awarded in 2020 but 
will not be completed until 2021. 


Accredited LRS. This work will audit ongoing or previous LRS accreditation efforts being performed 
by other DoD components (e.g., Navy, Army). As different DoD components work to secure the 
proper National Intelligence and Security Agency (NIST) and Defense Information Systems Agency 
(DISA) accreditations required to deploy different xAPI conformant LRS solutions across the DoD, 
USALearning will establish a repeatable process for integrating LRS solutions that conform to the 
Federal Information Security Management Act (FISMA) and Risk Management Framework (RMF) 
best practices, supports reciprocity between DoD components, and documents expectations for 
following common process, security controls, testing activities, and outcomes. This work supports 
the adoption of xAPI across the DoD services by creating policies, type accreditations, and a Security 
Technical Implementation Guide to support LRS accreditation across DoD networks. 


Culture Change for Competency-Based Learning. The TLA vision relies upon interoperable data 
across functional and organizational boundaries. This necessitates a paradigm shift and an 
investment in competency-based talent management. Competency-based learning emphasizes the 
demonstration of personnel capabilities rather than the measurement of instructional 
characteristics, better linking human performance to mission effectiveness. Key to the acceptance of 
this approach will be establishing common rules and workflows to support the development of 
frameworks and the sharing of data. 
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1.0 SCOPE 
1.1 Identification 


This document covers the objective end-state for Department of Defense (DoD) learning organizations 
migrating to Total Learning Architecture (TLA) compliant data and microservices. This is issued as the 0.9 
draft version of the document. Comments and feedback on all TLA artifacts are expected and updates 
will be made to each artifact as the TLA matures. Requirements will continue to evolve; therefore, these 
documents are iterative in nature. As the tools, technologies, and methodologies used in the future 
learning ecosystem change, TLA artifacts will be updated accordingly. 


1.2 System Overview 


The TLA defines a microservices based architecture for managing the data associated with learners, 
curricula, learning activities and content, competency tracking, and credential issuing. TLA policies 
include existing and emerging standards and specifications that enable interoperability between 
learning sites. They maintain education and training data to enable lifelong learning, and define business 
rules and governance strategies for managing the lifelong-learning data infrastructure. 


1.3 Document Overview 


The TLA Functional Requirements Document outlines the critical modes and states, capabilities and 
interfaces required for establishing a future learning ecosystem for the DoD. The TLA policy framework 
relies on existing and emerging specifications and standards to develop the data and services needed for 
the ledgering of performance data for “total learning” activities across the DoD human capital supply 
chain. 


The human capital supply chain is a complex system with inherent challenges to accommodating TLA 
operability. The specific composition and arrangement of technologies at any location will differ and 
change over time. Capabilities come not from individual installations or databases, but the connections 
between installations and the enterprise-level data sharing, collection, analysis, and dissemination that 
support the planning and controlling of human capital accession, detailing, and especially, education and 
training. 


This document introduces the overall requirements for elements of a TLA compliant installation. It 
provides detailed requirements for each of the service segments that enable the ledgering capabilities. 
The TLA architecture is derived from the four pillars of the integrated data strategy: 


e ALearner Profile to record learner credential history, aptitudes, local and global preferences, and 
local state -- which can be shared at the enterprise level (leveraging federated identity to protect 
privacy) to provide a complete portrait of human performance. 


e ACompetency Framework to define the elements, relationships, standards, contexts, and levels of 
mastery required to perform jobs and to certify the credentials used to define job placement. 


e ALearning Record Store to store the records of learning experiences that improve performance and 
operational data indicating effective learning transfer and impact on mission effectiveness. 


e An Experience Index to list and describe the activities that provide a context for a learning or 
assessment opportunity, and the content that is provided within that context. Activity-Content 
tuples are linked with competencies that they trigger into “experiences” which can be scheduled or 
launched for learners. 
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2.0 REFERENCED DOCUMENTS 


e |JEEE P9274.3.1 TLA Master Object Model Profile: included as Appendix B 
e |EEE P9274 Experience Application Program Interface (xAPI): https://site.ieee.org/sagroups-9274-1-1/ 
e Learning Resource Metadata Initiative (LRMI): https://www.dublincore.org/specifications/Irmi 
e JEEE P1482.20.2 Reusable Competency Definition Objects: 
https://standards.ieee.org/project/1484 20 1.html 
e cmi5 specification: http://aicc.github.io/CMI-5 Spec Current 


e OpenBadge: https://openbadges.org/ 

e Credential Transparency Definition Language: https://credreg.net/ctdl/handbook 

e Enterprise Learner Record Repository Metamodel: TBD 

e Learning Tools Interoperability (LTI): https://www.imsglobal.org/activity/learning-tools- 


interoperability 
e Open Identification (OpenID): https://openid.net/ 


3.0 REQUIREMENTS 


The requirements defined in this document specify the functions and interfaces that would be deployed 
at a given location to enable the ledgering and interoperability requirements that create the TLA 
capabilities. Migration from the legacy systems will occur in stages, as a key component of the overall 
TLA strategy is legacy recapitalization of a learning organization’s existing IT infrastructure. This means 
not all requirements need to be deployed in the initial rollout. The model showing TLA migration 
priorities is included in Section 4 of the main TLA report. 


3.1 Required Modes and States 


TLA compliant systems are intended to operate continuously. Servers deploying data and components 
should be selected for robust scalability and availability. Operations on TLA components happen in two 
main modes — maintenance mode and training mode. 


e Maintenance mode. The TLA compliant services and data provide a ledger of information about 
learners, the things these learners learn (competencies) and the way this learning occurs or is 
evaluated (activities and content). There is both a real-time and a static component to the data that 
provide this ledgering capability. Maintenance mode is concerned with the update of the static 
components that manage the introduction of new users, or movement of users between locations, 
the assignment and metadata attribution of new activities and content, and the import, export or 
modification of competency elements. 


e Training mode. The TLA compliant services working in conjunction to search static data and historic 
performance data, to plan and control new learning opportunities, collect performance data about 
them, and update learner state based on those data define the training mode. Training mode is 
continuously in operation. All learner data are not collected in real time but do have a quality of 
service associated with time of receipt and processing. 


Maintenance mode is essentially stateless at the system level, although individual databases have login, 
data entry, commit, and verification states depending on the technology used. The training mode 
similarly has component states based on the products being used. The entire federation is also viewed 
as stateless. A core 2019 research objective was the removal of the singleton state manager used to 
support the 2018 TLA Reference Implementation. 
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This decoupling was accomplished by modeling the learner state. Since the learner is the only 
component of any particular TLA instance that can be guaranteed present, we can standardize the 
behavior of event-driven services by documenting what occurs as the learner interfaces with each 
component in the system, independent of where that interaction occurs. We can then contextualize it 
within the services required for that learner state by generating xAPI statements to document each 
event. These statements are detailed in the Master Object Model (MOM) through the verb sequence 
used to model the learner state. Each MOM verb is generated as an activity stream signaling services to 
perform functions. 


Learning may unfold in a deliberate or informal fashion. In deliberate learning, the learner is explicitly 
organizing learning goals, building lists of activities into tasks to achieve those goals, and scheduling the 
performance of tasks through available resources. This represents the learner’s configuration of their 
learning environment. This configuration may happen exclusively in “edge devices” or in the interplay 
between edge devices and the core services. Activities are then launched by the TLA services and 
learning results are captured. 


Alternatively, learning may occur ad hoc, where an instrumented activity is experienced outside the 
context of a planned learning activity. In this case, the activity’s impact is still captured and 
contextualized. In either case, the evidence of the learning experience is evaluated against the records 
of competency and objective credentials, to include trust in the veracity of evidence collected. The 
learner’s competency and credential states are then “located” or updated in the globally discoverable 
records. This learner state cycle is depicted in Figure 1. 


Point of Entry for Deliberate Learning 


Q.Aa-Aa-A————— 
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Figure 1. Learner State as a Function of the Learning Lifecycle. Allowable learner state transitions that amplify this 
model are defined in the Master Object Model (MOM). The MOM captures as events the interactions between the 
learner and either edge or core devices and interfaces to trigger TLA services to act on learner data. 
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3.2 System Capabilities 


TLA compliant systems have a defined set of core services and core data stores. Each of these has 
associated requirements, but these requirements may be satisfied, or allocated to different 
components, if the functions and interfaces are satisfied. The 2019 TLA Reference Implementation uses 
a microservices architecture with streaming communication services that replaces the singleton-based 
point-to-point communications used in the 2018 Reference Implementation. 


3.2.1 Learning Event Management 


TLA compliant systems use different approaches for the goal setting, planning, scheduling, launching 
and capture of learning events. Learning events place a learner in an experience, composed of an 
activity, optionally viewing some content within the context of that activity, to achieve a learning goal or 
set of goals. The performance of the activity generates performance records, which are contextualized 
within the overall learning environment according to the MOM profile. 


Learning Event Management is most closely associated with the Learning Record Store (LRS) component 
of the core data, which is the server-side component of the xAPI specification. Learning Event 
Management requirements are presented in Table 1. Learning Event Management includes managing 
the learning path, setting goals, and validating resources available to conduct learning. 


Table 1. Learning Event Management Requirements. The Learning Event Manager is a set of services associated 
with goal selection, learning planning, scheduling of events and capture and contextualization of those events. The 


Learning Event Manager is associated with the Learning Record Store. 


Header Requirement Priority 
Learning Record Store 
Learning TLA compliant systems shall maintain a persistent storage of | 2019 
Record Store | learning activity records (i.e. LRS) 
Learning TLA compliant systems shall capture all xAPI statements 2019 
Record Store | generated from learning record providers 
Learning TLA compliant systems shall ensure that xAPI statements 2019 
Record Store | are complete and well formed 
Learning TLA compliant systems shall provide a mechanism for 
Record Store | administrators to purge old xAPI records 
Learning TLA compliant systems shall maintain a record of purges to 
Record Store | show that data has been altered 
Learning TLA compliant systems shall provide a mechanism to ensure | 2019 
Record Store | the integrity of xAPI data stored 
Learning TLA compliant systems shall allow storage of xAPI 2019 
Record Store | statements for the current user UUID stored as actor 
Learning TLA compliant systems shall allow use of filters on retrieving | 2019 
Record Store | xAPI data by Actor (user, user interest group), date/time, 

activity type (object), verb, user specified extension field 

values 
Learning TLA compliant systems LRS shall support federated data 2019 
Record Store | storage, search and retrieval between the noisy, 

transactional and Authoritative LRS 
Learning The Authoritative LRS shall be able to federate data from 2019 
Record Store | transactional LRS located in multiple enclaves 
Learning TLA compliant systems transactional LRS shall be sized to ELRR 
Record Store | support a 10-year digital data retention store of all evidence 
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Header Requirement Priority Comments 
Learning TLA compliant systems shall include a transactional LRS as 2019 
Record Store | part of core data that stores only data generated according 
to the TLA MOM profile (IEEE P9274.3.1) 
Learning TLA compliant systems shall have an Authoritative LRS that | ELRR 
Record Store | stores digitally signed xAPI statements of “conferral”, 
“qualification” and “certification” for competency 
assertions. 
Learning TLA compliant systems shall preserve the traceability ELRR 
Record Store | between evidence, assertions, 
qualification/certification/conferrals and globally 
discoverable digital badges for credentials 
Learning TLA compliant systems shall use "noisy" LRS to segregate 2019 
Record Store | data for device specific profiles 
Learning Noisy LRS profiles shall comply with IEEE P9274.2.1 2019 
Record Store 
Learning TLA compliant systems shall identify a "boundary" learning | 2019 
Record Store | record provider that conforms to the TLA MOM for all edge 
devices generating learning evidence (operational data 
sources or learning activities) 
Learning TLA compliant systems shall identify an Authoritative LRS ELRR 
Record Store | for storage of conferred user credentials 
Learning The LRS shall comply with the server-side component of the | 2019 
Record Store | xAPI specification (IEEE P9274) 
Manage Learning Path Logic 
Manage Learning Event Management service shall manage progress | 2019 
Learning Path | through sub-goals until selected goal is achieved 
Logic 
Manage Learning Event Management service shall be able to 2019 
Learning Path | schedule launch of single experiences from the learning 
Logic experience catalog 
Manage Learning Event Management service shall be able to launch | 2019 
Learning Path | a course from a catalog 
Logic 
Manage Learning Event Management service shall be able to launch | 2019 
Learning Path | from a user or observer/instructor/controller/ supervisor 
Logic (OICS) curated content list 
Manage Learning Event Management service shall support courses 2019 
Learning Path | of a single content resource, or multiple resources 
Logic 
Manage Learning Event Management service shall support default 2019 If SCORM and more 


Learning Path 
Logic 


paths through multi-asset course or content set 


than one SCO, using 
S&N tags 


Manage Learning Event Management service shall send selected 2019 
Learning Path | courses or experiences to the learner profile as pending 

Logic tasks 

Manage Pending tasks shall include a suspense date 2019 


Learning Path 
Logic 
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Header Requirement Priority Comments 

Manage Learning Event Management service shall generate a 2019 

Learning Path | "scheduled" MOM message when experiences or courses 

Logic are listed as pending in the learner profile 

Manage Learning Event Management service shall support user 2019 If SCORM and more 

Learning Path | selected paths through a multi-asset course or content set than one SCO, then 

Logic user select sequencing. 
Courses may be 
created w/o SCORM 
packaging, and using 
their own defined logic 
in which case, each 
content element is 
registered 
independently 

Manage Learning Event Management service shall determine if 

Learning Path | courses require permission to launch or schedule 

Logic 

Manage Learning Event Management service shall generate a 

Learning Path | "requested" xAPI message when experiences or courses 

Logic require OICS approval 

Manage Learning Event Management service shall allow the OICS to 

Learning Path | approve attendance in a course or experience 

Logic 

Manage Learning Event Management service shall allow the OICSto | 2019 

Learning Path | assign a learner or identity group attendance in a course or 

Logic experience 

Manage Learning Event Management service shall generate a 2019 

Learning Path | "directed" xAPI message when experiences or courses were 

Logic directed by the OICS 

Manage Learning Event Management service shall allow for machine |Outyear | "recommenders" 

Learning Path | learning to create a list of applicable content customized to 

Logic the individual learner and current learner state 

Manage Learning Event Manager service shall generate a 2019 

Learning Path | "socialized" xAP| message when a curated list is shared 

Logic between learners 

Manage Learning Event Management service shall allow learners to | 2019 

Learning Path | select from their assigned or created curated lists 

Logic 

Manage Learning Event Management service shall be able to capture | 2019 

Learning Path | registered learning device that load or launch content 

Logic 

Manage Learning Event Management service shall generate a 2019 

Learning Path | "captured" xAPI message when unscheduled experiences or 

Logic courses are launched 

Manage Learning Event Management service shall generate an Objective for 2019 

Learning Path | "augmented" xAPI message if a selected goal is already a 

Logic demonstrated competency 

Manage Learning Event Management service shall be able to launch = | 2019 Content server 


Learning Path 
Logic 


selected activities on the same client as the LEM is accessed 
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Header Requirement Priority Comments 

Manage Learning Event Management service shall be able to launch | 2019 "in the wild" 
Learning Path | selected activities on remote client devices that have been 

Logic registered with the network 

Manage Learning Event Management service shall be able to 2019 Decouple "launching" 


Learning Path 
Logic 


associate experiences that were launched locally with the 
appropriate task if they were previously scheduled 


from "scheduling" 


Manage Learning Event Management service shall be able to verify Objective for 2019 
Learning Path | that an activity has closed out with completed, abandoned 

Logic or terminated 

Manage Learning Event Management service shall generate the Objective for 2019 


Learning Path 
Logic 


"abandoned" xAPI message after activity timeout 


Goal Setting 

Goal Setting | Learning Event Management service shall allow selection of | 2019 
one or more goals at an arbitrary level from within a 
selected competency framework 

Goal Setting | Selected learner goals shall be assigned a priority 2019 

Goal Setting | Selected leaner goals shall be stored persistently until 2019 
achieved 

Goal Setting | Learning Event Management service shall provide a default | 2019 Key finding and update 
path through the selected competency graph as a sequence from 2018 
of sub-goals 

Goal Setting | Learning Event Management service shall enable a user 2019 Key finding and update 
defined or configured path through the sub-competency from 2018 
goals 

Goal Setting | Learning Event Management service shall enable Outyear | Recommender (macro 
algorithmically defined or configured path through the sub- adaptation) 
competency graph 

Goal Setting | Learning Event Management service shall allow an OICS to 2019 Stub class for testing, 
assign a goal to a learner or identity group pending CASS 

Goal Setting | Learning Event Management service shall generate a 2019 Key finding and update 
"projected" xAPI message when the learner or adaptation from 2018 
service created the sub-goals path 

Goal Setting | Learning Event Management service shall generate an 2019 Key finding and update 
"organized" xAPI message when the learner used the from 2018 
default sub-goals path 

Goal Setting | Learning Event Management service shall filter the 2019 Key finding and update 
experience catalog by selected goals and subordinate sub- from 2018 
goals when active 

Goal Setting | Learning Event Management service shall allow the userto | Optional | Key finding and update 
select single level or fully recursive filters for experience from 2018 
selection 

Goal Setting | Learning Event Management service shall generate a 2019 
"directed" xAPI message when the OICS passes a goal to the 
learner or identity group 

Goal Setting | Learning Event Management service shall generate a 2019 


"planned" xAPI message when a leaner has selected goals 
and/or sub-goals 


Error Trapping 
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Header Requirement Priority Comments 
Error Learning Event Management service shall be able to identify 
Trapping incoming xAPI statement with an actor that is not a valid 
user, registered component, or identity group 
Error Learning Event Management service shall be able to identify 
Trapping if an incoming xAPI statement is not well formed 
Error Learning Event Management service shall be able to identify 
Trapping that an incoming xAPI statement is not from a registered 
device 
Error Learning Event Management service shall be able to identify 
Trapping that an incoming xAPI statements references an invalid 
catalog item 
Error Learning Event Management service shall generate an 
Trapping administrator alert if invalid xAPI statement is received 
MOM Profile 
MOM Profile | TLA compliant enclave and federation shall be able to 2019 
process learner state IAW the TLA MOM (IEEE P9274.3.1) as 
received from edge devices 
MOM Profile | TLA compliant enclave and federation shall be able to 2019 
process learner state IAW the TLA MOM (IEEE P9274.3.1) as 
received from the alert and notification system 
MOM Profile | TLA compliant enclave and federation shall be able to 2019 
process learner state IAW the TLA MOM (IEEE P9274.3.1) as 
received from a user interface 
MOM Profile | TLA compliant enclave and federation shall be able to 2019 
process learner state IAW the TLA MOM (IEEE P9274.3.1) as 
detected from interaction with TLA data resources and 
services 
MOM Profile | TLA compliant core services and data shall process 2019 
performance evidence from actionable information IAW the 
TLA MOM (IEEE P9274.3.1) 
MOM Profile | TLA compliant system shall process learner career state IAW | 2019 
the TLA MOM as received from user interfaces 
MOM Profile | TLA compliant system shall process learner career state IAW |Outyear | Requires a future 
the TLA MOM as received from federated HR systems interface to M&P 
systems 
xAPI Profiles and Fields 
xAPI Profiles | The xAPI profiles of TLA compliant edge systems shall IAW P9274.2.1 
and Fields include templates for all learning content, activity, and 
experience types applicable to the federate instance 
xAPI Profiles | The xAPI profiles of TLA compliant edge systems shall IAW P9274.2.1 
and Fields include a complete object life cycle (from requirement, to 
selection, launch, work, and closeout) for each training 
technology type 
xAPI Profiles | TLA compliant edge systems shall include an xAPI profile Profile 
and Fields validation server to ensure compliance with the profile Server 
xAPI Profiles | TLA compliant core systems shall include a TLA MOM xAPI Profile 
and Fields profile validation server to ensure compliance with the Server 
profile 
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Header Requirement Priority Comments 
xAPI Profiles | TLA compliant systems xAPI profile shall include data Profile 
and Fields elements required to audit evidence of assertions of Server 
competence 
xAPI Profiles | TLA compliant systems xAPI profile shall include data Profile 
and Fields elements to specify context under which a work event was __| Server 
experienced 
xAPI Profiles | TLA compliant systems xAPI profile shall include data Profile 
and Fields elements to specify context under which an assessment was | Server 
evaluated 
xAPI Profiles | TLA compliant systems xAPI profile shall include data Profile 
and Fields elements to specify standard context under which an OJT Server 
event was experienced 
xAPI Profiles | TLA compliant systems xAPI profile shall include data Profile 
and Fields elements to specify areas not achieved during exams (i.e. Server 
grade<100%, what was missed?) 
xAPI Profiles | TLA compliant systems shall include verification against the | Profile 
and Fields profile conformance suite as part of enclave deployment or | Server 
update 
xAPI Profiles | TLA compliant systems shall include verification against the | Profile 
and Fields profile conformance suite as part of federation Server 
development and edge device registration 
xAPI Profiles | TLA compliant system xAPI profiles shall include fields for Profile 
and Fields simulator and lab exercises to capture details of learner’s Server 
role, actual scenarios employed, and competencies 
triggered as paradata 
Resource Validation 
Resource Learning Event Management system shall be able to verify 2019 
Validation availability of resources prior to launching event 
Resource Resources listed in the Experience Index shall include valid 2019 
Validation URL and available resources for web content, whether 
internal or external to the enclave 
Resource Resources in the Experience Index shall include a digital OPSEC consideration. 
Validation verification method when the resource is stored external to eHelm would satisfy 
the enclave 
Resource Resources listed in the Experience Index shall include Outyear 
Validation consumables, computational assets, rooms, 
instructor/facilitators, and instrumentation 
Resource TLA compliant systems shall be able to schedule constrained |Outyear | Probably interface to 
Validation learning resources another system or set 


of systems 


3.2.2 Activity Registry and Resource Management 


The Activity Registry and Resource Manager is associated with the creation, review, update and deletion 
of learning experiences, which are composed of activities, content, and metadata that combine to 
describe each experience. These metadata also define the competency alignment of the experience, its 
nature and modality, and fitness for use in an instructional, evaluative, or heutagogical setting. The 
metadata model for the 2019 TLA Reference Implementation is based on a modified version of the 


Learning Resources Metadata Initiative (LRMI) published by Dublin Core Initiative. 
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The Activity Registry and Resource Management services also support an Enterprise Course Catalog 
(ECC) capability. The actual composition of the Activity Registry and Resource Management services may 
change from location to location but federate to the ECC. The Activity Registry function is most closely 
associated with the Experience Index in the TLA data strategy. The Activity Registry function also 
includes device registration, such that devices are authorized delivery mechanisms for activities. The 
computational node generating the xAPI stream is the registered device, so if it is a server (like a 
traditional Learning Management System) it will handle its own client connections and only the server 
will be registered as an activity provider for purposes of TLA device registry. 


Device registry works with back-end services (identity and virtualization management) for security and 
integrity of data generated and processed from mobile platforms or other “ubiquitous learning devices” 
(which may include instrumented systems, QR Code readers, and any number of future technologies). 
Specific requirements are listed in Table 2. 


Table 2. Activity Registry and Resource Management Requirements. 


Header 


Requirement 


Enterprise Course Catalog (ECC) 


| Priority Comments 


Enterprise TLA compliant enclave shall provide a consolidated catalog of | ECC Simple features as 
Course Catalog | courses and registered content available for learning applicable for 2019 
Enterprise The ECC shall list courses by course name and supporting data | ECC Simple features as 
Course Catalog applicable for 2019 
Enterprise Common Course Supporting data shall include service unique | ECC Simple features as 
Course Catalog | identifier, provenance and quota control information, applicable for 2019 
availability, points of contact, registration and pre-requisite 
information 
Enterprise The ECC shall indicate courses required for regulatory ECC Simple features as 
Course Catalog | compliance applicable for 2019 
Enterprise The ECC shall have a mechanism to discover all enclaves with | ECC Simple features as 
Course Catalog | Enterprise Course Catalog data applicable for 2019 
Enterprise Users shall be able to access the Enterprise Course Catalog ECC Simple features as 
Course Catalog | from any TLA compliant enclave applicable for 2019 
Enterprise TLA compliant enclaves shall be able to share applicable ECC Simple features as 
Course Catalog | Enterprise Course Catalog data applicable for 2019 
Enterprise Updates to the course catalog shall generate a notification to | ECC 
Course Catalog | associated user interest group(s) 
Enterprise The ECC shall be able to add additional field or enumerated ECC 
Course Catalog | values to display based on the decisions of governance boards 
Enterprise The ECC shall require identity credentials for access to read, ECC 
Course Catalog | update, delete, or create entries 
Enterprise The ECC shall require identity credentials for access to modify | ECC 
Course Catalog | the reporting of constituent Experience Index course data 
Enterprise The ECC shall be able to list associated media for courses that | ECC 
Course Catalog | may be shared, including simulation scenarios, part task 
trainers, IETMS and digital libraries of electronic publications, 
computer-based training aids, and others 
Enterprise The ECC shall be able to accommodate service/agency specific | ECC 
Course Catalog | course identification numbers 
Enterprise The ECC shall be able to display information reported from ECC 
Course Catalog | financial control data sources pertaining to course 
provenance 
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Enterprise The ECC shall be able to display information reported from ECC 
Course Catalog | material control data sources pertaining to shareable resource 
inventory 
Enterprise Users shall be able to filter returned courses in the Enterprise | ECC 
Course Catalog | Course Catalog based on selection of any of the indicated 
attribute field enumerated values 
Enterprise The ECC shall integrate with the alert and notification system | ECC 
Course Catalog | to send updates or new course listings to select interest 
groups 
Enterprise The ECC shall pull data from connected experience indices ECC 
Course Catalog 
Enterprise The ECC shall be able to synchronize with an authoritative ECC 
Course Catalog | source for competency and credential data to populate 
educational alignment fields 
Enterprise The ECC shall support mapping data in the LOM and LRMI ECC 
Course Catalog | formats as appended for TLA 
Enterprise The ECC shall be able to return and filter results within five ECC 
Course Catalog | seconds 
Enterprise The ECC shall be able to resolve missing or blocked Experience | ECC 
Course Catalog | Index resources without a critical halt 
Enterprise The ECC shall generate an alert to an identified administrator | ECC 
Course Catalog | if an assigned Experience Index resource times out during 
connection attempts 
Experience Index 
Experience The Experience Index shall distinguish between data that is 2019 DIV2 
Index owned by an instance/enclave (authoritative source) and that 
which is copied to an instance/enclave from the authoritative 
source 
Experience The Experience Index shall distinguish between formal course, | 2019 DIV2 
Index supporting content (assets for course) and ancillary activities 
and content (not associated with a course) 
Experience The Experience Index shall include activities that are digitally | 2019 DIV2 
Index instrumented contexts under which learning content or in situ 
tasks can be experienced (e.g. simulators, LMS, readers, 
mobile devices) 
Experience The Experience Index shall include content and its associated | 2019 DIV2 
Index activity or activities in the form of digital assets that support 
the experience (e.g. eBooks, scenarios, SCORM and cmi5 
packages, Portable learning device corpus) 
Experience The Experience Index shall allow for experiences composed of | 2019 DIV2 
Index activities without content 
Experience The Experience Index shall list applicable or allowable 2019 DIV2 
Index activities for use of content 
Experience The Experience Index shall include metadata for each activity, | 2019 LRMI+ until XI/ECC 
Index content and experience that describes its educational purpose metamodel 
as intended established 
Experience The Experience Index shall include metadata for each activity, | 2019 DIV2 
Index content and experience that describes its provenance and 
authority, its creation and version information, and its 
nomenclature 
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Experience The Experience Index shall include metadata for each activity, | 2019 DIV2 
Index content and experience that describes an object handle for 
use in xAPI statements 
Experience The Experience Index shall include metadata for each activity, | 2019, DIV2 
Index content and experience that describes details regarding its but data 
modality, instructional style and impact on learner cognitive labeling 
or physical attributes such that two experiences otherwise and ECC 
labeled identically can be evaluated and prioritized for an 
individual learner 
Experience The Experience Index shall allow for the listing of a single 2019 Supports 
Index SCORM/cmi package as a collection of associated recapitalization and 
competencies and metadata for a single experience "object ownership" 
Experience The Experience Index shall allow for the listing of a 2019 Supports 
Index decomposable SCORM/cmi package as a collection of recapitalization and 
associated competencies and metadata for each uniquely "object ownership" 
launchable portion of the experience 
Experience The Experience Index shall allow for the creation of a 2019 Supports 
Index hierarchical course from any allowable combination of recapitalization and 
activities and content which have not been packaged using "object ownership" 
SCORM or cmi5 
Experience The Experience Index shall include named "content set" lists | 2019 Supports 
Index of experiences compiled by a learner or OICS recapitalization and 
"object ownership" 
Experience Experience Index Content Set lists shall by default be visible to Supports 
Index the learner that created them recapitalization and 
"object ownership" 
Experience OICS shall automatically share Experience Index content set Data privacy 
Index lists with learners in their assigned identity groups settings 
Experience Learners shall be able to select Experience Index content lists Akin to sharing a 
Index to be shared with other learners or OICS playlist 
Experience The Experience Index shall also register applicable OJT/work Required for full 
Index experiences as activity types competency-based 
tracking in maturity 
level three. May be 
federated such that 
only local work 
opportunities are 
managed within a 
given instance 
within a local 
version of the 
Experience Index 
Experience The Experience Index shall be able to register local devices as 
Index activity types 
Experience The Experience Index shall be able to list one or more other 
Index resources for an activity 
Experience The Experience Index shall be able to map any locally loaded 
Index content on devices as unique experiences 
Experience The Experience Index shall include ordered sets of 
Index subordinate activities and content as a course 
A-12 | 2019 Total Learning Architecture Report — Appendix A - TLA Functional Requirements Document 


Header 


Requirement 


| Priority Comments 


Experience The Experience Index shall include ordered sets of 
Index subordinate activities and content as a user curated list 
Experience Learners may share their curated lists with other users 
Index 
Experience OICS may create curated lists and direct all or a subset of their 
Index learners to complete the list 
Search Function 
Search The search function shall allow filtering and search of 2019 Establish indices 
Function elements in the Experience Index by job, credential, 
competency defined at any level, level of mastery, activity 
type, authority 
Search The search function shall generate a "explored" xAPI message 
Function if a search is similar to recent content searches 
Search The search function shall generate a "clarified" xAPI message 
Function if a search is similar to recent competency goal support 
searches 
Resource Management 
Resource TLA compliant systems shall ensure resources (classroom Probably a separate 
Management materials, instructors, facilities, server resources) necessary to class scheduling app 
schedule content are available 
Resource TLA compliant systems shall reserve the required resources Probably a separate 
Management when a content session is requested class scheduling app 
Resource If the resource is an OICS, a notification shall be sent to that Probably a separate 
Management system user of their schedule class scheduling app 
Resource TLA compliant systems shall manage facility/OICS resource Probably a separate 
Management requests in batches tied to registrar end date class scheduling app 
Resource Computational resources shall be scheduled when required to May be offline 
Management run simulations or host content process 
Resource TLA compliant systems shall verify on-line (WWW) and digital | 2019 
Management in-house content is available including the server up and file 
path is valid 
Resource TLA compliant systems shall provide a binary level content Interface 
Management | verification method for registered content which is external to requirement- 
the enclave and on unprotected resources (e.g. WWW) possibly eHelm 
Activity Registry 
Activity The Activity Registry shall generate an alert to the Interface to alert 
Registry Content/Competency Management Service s advertising and notification 
updates system 
Activity The Activity Registry shall support definition of simulations or | 2019 
Registry other laboratory exercises within an overall curriculum 
Activity The Activity Registry shall support definition of instructor led | 2019 
Registry training curricula 
Activity The Activity Registry shall allow for creation of curated lists 2019 
Registry 
Activity The Activity Registry shall associate additional activities and 2019 Supports ILT/ 
Registry supporting content with formal courses (e.g. for ancillary blended learning 
teaching materials, extra learning or tutoring, lesson plans curation process for 
and trainee guides) ancillary teaching 
aids 
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Activity The Activity Registry shall generate a "curated" TLA MOM 2019 
Registry message when experiences or courses were grouped in a 
content set list by the OICS or learner 
Activity The Activity Registry shall support decomposition of simulator 2019 Objective 
Registry content to the lowest tracked level of execution (e.g. scenario, 
MSEL, Initial Condition Set, etc.) 
Activity The Activity Registry shall be able to register web content 2019 Objective 
Registry external to the TLA enclave as an experience 
Activity External content registered as an experience shall otherwise 
Registry comply with cybersecurity restrictions for the appropriate 
security level for that security level 
Activity The Activity Registry shall be able to register a content 2019 
Registry management system internal to the TLA enclave as an 
experience 
Activity The Activity Registry shall generate an alert shall require an Interface to alert 
Registry admin, course manager, or curriculum manager to authorize and notification 
content for use system 
Activity The Activity Registry shall generate an alert for content that is | Outyear | Requires interface 
Registry a normative reference which has as updated configuration to PDSS systems 
Activity The Activity Registry shall allow delisting of experiences from | 2019 
Registry the Experience Index 
Activity The Activity Registry shall allow content and Competency 2019 
Registry Management Service s to update metadata associated with 
content, activity and experiences 
Activity The Activity Registry shall be able to filter, and search based Outyear | Interface to CMS - 
Registry on educational alignment objects which have been updated cascading impact of 
changes 
Activity The Activity Registry shall allow OICS to list simulators, labs, 2019 DIV2 
Registry OJT, EPSS, and user defined work experiences to be registered 
as valid activity types 
Activity Updates to owned experiences shall generate alerts to 
Registry federated systems that share experiences 
Activity The Activity Registry shall maintain a log of all changes by 
Registry source, date, record affected, old and new version 
Activity The Activity Registry shall be able to import a named content 
Registry set list from a federated Experience Index 
Device Registration 
Device The Activity Registry shall be able to register authorized 2019 Objective 
Registration devices as activities within a federation 
Device Registered devices shall include operational data sources or Outyear | As applicable 
Registration middleware systems (i.e. anything that will generate xAPI 
statements for the transactional LRS) 
Device Registered devices shall include handheld devices 2019 
Registration 
Device Registered devices shall include content repositories or 2019 
Registration middleware (i.e. anything that will generate xAPI statements 
for the transactional LRS) 
Device Registered devices shall include learning management servers | 2019 
Registration (i.e. anything that will generate xAPI statements for the 
transactional LRS) 
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Device Registered devices shall include any other client device, 2019 e.g. GIFT 
Registration computer or middleware application that that will generate 
xAPI statements for the transactional LRS. 
Device Device registration shall expose endpoints for xAPI messages 2019 Objective 
Registration and launch commands 
Device Device registration shall include the use of LT! descriptors for 2019 Objective 
Registration tools 
Device Device Registration shall include Synchronization of user data 2019 Objective 
Registration to the enclave anonymization tokens 
Device Device registration shall include synchronization of device 2019 Objective 
Registration content to Experience Index object handles appropriate 
Device Device registration shall include a dynamically assignable 
Registration multi-factor authentication (MFA) process. 
Device Registered devices shall be able to map local identity to the 
Registration centrally managed identity in the TLA core 
Device Connected devices shall support remote launching 
Registration 
Device Connected devices shall provide time out alerts to 
Registration administrators, learners and OICS, and error handling if they 
are unavailable for a remote launch 


3.2.3 Competency Management 


Competency management includes the services associated with defining competency frameworks that 
define the required levels and behaviors of human performance with the jobs and credentials that 
require that knowledge, skills, abilities and other (KSAO) behaviors. Competency objects are defined by 
the reusable competency definition model and a directed acyclic graph (DAG). This type of hierarchy 
allows for multiple parents, enabling sharing of competency fragments to show how knowledge gained 
in one discipline may bolster capability in another. Competency management includes the maintenance 
of frameworks and the evaluation of performance evidence to make assertions of individual 
competence. Competencies are generated from local materials in accordance with the reusable 
competency definition (RCD) objects. 


Competency and credentials are managed at the DoD enterprise level and require a digital trust system. 
This system of digital trust provides auditability, transparency, non-repudiation, privacy and data 
integrity for the data associated with capable manpower as a critical national capability. The digital trust 
system includes encrypted, digitally signed, credential management and a global system of governance 
that allows for customization. It also facilitates discoverability and sharing of common components 
while ensuring traceability to a single owner for configuration management. 


Learners encounter learning activities that present opportunities for instruction and assessment. These 
are aligned with competencies using alignment objects that are part of the LRMI specification. As a 
learner interacts with each learning activity, they generate evidence of competence. The evidence is 
evaluated based on a trust value and on an efficacy value. Evidence is normalized according to the cmi5 
profile and the contextualization of the TLA MOM. Evidence of competence, along with assertions of 
competence based on evaluation of this evidence, are stored as xAPI messages in the transactional 
Learning Record Store (LRS). This maintains the “chain of evidence” for competency. A Competency 
Management System evaluates evidence from the transactional LRS to predict learner mastery. 
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These mastery estimates are housed in the form of a credential and are stored within the Authoritative 
LRS as xAPI statements, along with the digital signature of a trusted individual who has been 
empowered to make the conferral of credential. This maintains the “chain of custody.” Authoritative LRS 
messages are linked to an Open Badge or CTDL portable badge version of the credential in the learner 
profile database, which can be shared across the enterprise through the Enterprise Learner Record 
Repository. This chain of evidence and chain of custody is shown in Figure 2. 
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Figure 2. Chain of Custody for Competency and Credential Management. Adjudicated performance records are 
collected in the transactional LRS, along with calculated assertions of competence and contextualization data. 
These xAPI present an evidentiary chain for supporting the award and review of credentials, which are stored as 
digitally signed xAPI within the Authoritative LRS. Learner State data, local and global learner preferences, and 
universally discoverable OpenBadge3 versions of credentials located in the learner profile database, and all data 
owners with credential records for a leaner are maintained and visible through the Enterprise Learner Record 
Repository. 


Table 3. Requirements for the Competency and Credential Management Services. 


Header Requirement Priority Comments 
Learner Profile 


Learner Profile | The local learner profile shall be developed consistent 
with the TLA learner profile metamodel (Spec TBD) 


Learner Profile | The learner profile shall link back to an authoritative 2019 LP is broken out separately 
identity management service for PPI (personal data: at maturity level three, at 
name, rank, SSN, address, phone, UIC) one and two is probably 

student management 
service of LMS, or HR 
system organic capability 

Learner Profile | The learner profile shall use the internally generated 2019 
anonymization token for storing user data 

Learner Profile | The learner profile shall maintain a list of asserted 2019 
competencies and effective dates 

Learner Profile | The learner profile shall maintain a list of conferred 2019 LP is broken out separately 
credentials, CEU state, and effective dates at maturity level three, at 


level one and two is 
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probably student 
management service of 
LMS, or HR system organic 
capability 


Learner Profile 


The learner profile shall maintain a list of authorized 
access roles 


2019 


May be tied to identity 
management system or 
back-end security services 


Learner Profile 


The learner profile shall be able to store user specified 
attribute data defining learner preferences 


2019 


Learner Profile 


The learner profile shall maintain a change log of 
updates to the profile 


2019 


Learner Profile 


The universal learner profile shall support deployment as 
a federation of multiple local learner profiles, at multiple 
levels of security 


ELRR 


Learner Profile 


The learner profile shall be able to link logically lower 
level enclave data with higher level data to form a 
complete picture of performance 


ELRR 


Learner Profile 


The learner profile shall be able to link digitally lower 
level enclave data with higher level data to form a 
complete picture of performance 


ELRR 


Only where CDS available 


Learner Profile 


The universal learner profile shall be able to follow the 
learner/user through multiple assignments 


ELRR 


Learner Profile 


The learner profile shall store current performance goals 
and sub goals as job, duty, gig, competency, or credential 
hierarchies or elements of hierarchies 


2019 


Learner Profile 


The learner profile shall store current learner state and 
assigned tasks/experiences 


2019 


Modified or informed by 
ELRR 


Learner Profile 


The learner profile shall indicate when tasks have been 
satisfied by events 


2019 


Learner Profile 


The learner profile shall allow for the creation, retrieval, 
update and deletion of learner records 


2019 


Learner Profile 


Deleted learner records shall be recoverable/auditable 


Learner Profile 


The learner profile shall provide a mechanism to ensure 
credentialing and de-credentialing comes from a non- 
repudiable and authoritative source 


May be separate process. 
Can CASS process the 
whole conferral review 
process? Use “signatures” 
as a class of evidence — 
from a non-repudiable 
approval tracking system? 


Learner Profile 


Individual learner profile records shall enable an 
administrator to conduct a full record purge after a 
specified period 


Learner Profile 


The learner profile shall indicate the current possible 
career trajectories (badges, jobs, etc.) 


Future interface 
requirement to M&P 
systems 


Learner Profile 


The learner profile shall maintain a mechanism to 
prevent hacking/loss of data integrity 


ELRR 


Simple for 2019 


Learner Profile 


The learner profile shall be able to assign learner 
personas 


ELRR 
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Learner Profile | The learner profile shall maintain a record of aptitude 2019 Include airman learning 
definitions (as locally defined name value pair sets) record type fields, as well 
as anything required to 
support adaptation 
algorithms of decision 
support 
Learner Profile | The learner profile shall integrate with the competency | 2019 This is part of the 
and Credential Management Services definition for level three 
maturity 
Learner Profile | Roles and Personae shall depict an arbitrary level of Simple for 2019 
hierarchical specificity 
Learner Profile | The learner profile shall store all applicable personae or |ELRR 
roles associated with the person pertaining to a unique 
portion of their total learning 
Learner Profile | The learner profile shall support global discovery and 2019 This is part of definition of 
federated data structures between enclaves level three maturity 
Learner Profile | The learner profile shall maintain an auditable log of ELRR May be separate service 
changes 
Learner Profile | The learner profile shall provide a digital export of ELRR May be separate service 
credentials for civilian portability 
Learner Profile | The learner profile shall allow for deletion of records ELRR May be separate service 
from searches by an administrator 
Learner Profile | The learner profile shall allow for “hiding” (non- ELRR 
permanent deletions) of user data from searches and 
displays by an administrator 
Learner Profile | The learner profile shall provide the unit identification ELRR 
data for where credentials were awarded to the 
enterprise level record repository 
Learner Profile | The learner profile shall provide unit location data for ELRR 
credentials being awarded 
Learner Profile | The learner profile unit location data shall include ELRR 
location data for mobile, deployed or mobilized units 
Manage Competency Framework 
Manage Competency Frameworks shall be developed IAW IEEE CASS 
Competency 1484.20.1 RCD model 
Framework 
Manage The Competency Management Service shall maintain a CASS Likely included as part of 
Competency list of jobs/duties required of each user role within the HR system for maturity 
Framework organization level one and two, 
transition to Competency 
Management Service for 
level three 
Manage The Competency Management Service shall store the CASS Likely included as part of 
Competency knowledge, skills, abilities, and other (KSAO) behaviors HR system for maturity 
Framework required to perform a job or duty level one and two, 
transition to Competency 
Management Service for 
level three 
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Manage Each KSAO shall include relationships between CASS Content Aggregation 

Competency competency definition objects and associated Model (CAM) within LMS 

Framework context/conditions and standards courses maintains a 
hierarchy for maturity 
levels 1 & 2, but this will 
be replaced with directed 
acyclic graph of 
competency objects and 
evidence maps for level 3 

Manage The context and standards under which competencies CASS 

Competency | were acquired shall support determining fitness of the 

Framework person for a specific job or employment 

Manage The Competency Management Service shall define CASS Levels of mastery might be 

Competency related competency objects (cognitive, psychomotor, implicit in LMS course 

Framework affective, social, and metacognitive domains, standards, structures but may 

and context/conditions) at multiple levels of mastery become a lower level 

attribute in maturity level 
three competency 
frameworks 

Manage The Competency Management Service shall specify the | CASS Included in HR system 

Competency | competencies and level of mastery required for each mapping to required 

Framework job/duty courses in maturity level 
one and two 

Manage The Competency Management Service shall be able to CASS May have additional 

Competency | distinguish between qualification, proficiency, and categories specified — 

Framework mastery shows currency of 
capability 

Manage Credentials defined for a job/duty shall link to CASS Credentials may include a 

Competency competency objects required to perform a job/duty traditional “degree” or 

Framework “certificate” as well as 
“necessary qualifications” 
to do the job 

Manage The Competency Management Service shall be able to CASS 

Competency update any inference weights 

Framework 

Manage The Competency Management Service shall generate an | CASS 

Competency “inferred” message if weights have been updated 

Framework 

Manage The Competency Management Service shall be able to CASS Outyear requirement 

Competency import operational performance data to validate the 

Framework elements and weights of a DAG 

Manage The Competency Management Service shall generate a CASS Outyear requirement 

Competency “validated” xAPI message if the weights have been 

Framework updated 

Search Function 

Search The Competency Management Service shall allow a CASS* Might be separate service, 

Function learner to search on all credentials associated with a job support LEM 

Search The Competency Management Service shall allow a CASS* Might be separate service, 

Function learner to search on all competencies associated with a support LEM 


job 
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Search The Competency Management Service shall allow a CASS* Might be separate service, 
Function learner to search on all competencies associated with a support LEM 
credential 
Search The Competency Management Service shall allow a CASS* Might be separate service, 
Function learner to search on all sub-competencies associated support LEM 
with a competency 
Search The Competency Management Service shall allow a CASS* Might be separate service, 
Function learner to search on changes to competency framework support LEM 
by date 
Search The Competency Management Service shall allow a CASS* Might be separate service, 
Function learner to search by competencies with different levels support LEM 
of mastery (i.e. what jobs are associated with each level, 
and what standards and context applies) 
Search The Competency Management Service shall generate a | CASS* Might be separate service 
Function “clarified” xAPI message if a competency is selected that 
reinforces a recently completed experience that is not on 
the current goal-activity plan 
Search The Competency Management Service shall generate a | CASS* Might be separate service 
Function “augmented” xAPI message if a competency is selected 
that reinforces a recently completed competency that is 
not on the current goal-activity plan 
Search The Competency Management Service shall allow a CASS* Might be separate service 
Function learner to search on competency owner 
Search The Competency Manager Service shall export search CASS* Might be separate service- 
Function results as a serialized array of competency objects used to support LEM goals 
Update Learner Competency 
Update The Competency Management Service shall generate CASS Supported in transition 
Learner assertions of competence based on evidence of mastery from course-based to 
Competency competency-based 
Update The Competency Management Service shall maintain an | CASS May use linked data to LRS 
Learner evidentiary history of local training events/exercises at maturity level three 
Competency _| attempted and completed, as well as scoring data when the LP is broken out. 
Update Evidence of mastery shall include feeds from any CASS This is preserved in the LRS 
Learner instrumented digital learning device that can generate and is key to the first 
Competency | xAPI migration level 
Update LRS shall federate data at TLA compliant core boundary | CASS 
Learner by using TLA MOM verbs for Learner record provider 
Competency state (equivalent to cmi5 states) 
Update The Competency Management Service shall process CASS Real world scenarios may 
Learner cascading evidence chains through associated indicate mastery of 
Competency | competency frameworks (showing all competencies knowledge that is related 
demonstrated by the evidence) to multiple competencies 
(e.g., many to many 
relationships instead of 
the one to many 
hierarchies used in 
SCORM’s CAM, which 
mirrors course structures 
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Update The Competency Management Service shall be able to CASS This is analogous to testing 
Learner calculate progress toward a related credential as a out of scholastic 
Competency __| sequence of demonstrated competencies requirements. 
Update The Competency Management Service shall be able to Outyear | Future M&P system 
Learner import learner Career state intersection, or this 
Competency information can be input 
manually 
Update The Competency Management Service shall be able to CASS 
Learner calculate progress toward competencies not associated 
Competency | with a credential 
Update The Competency Management Service shall determine CASS This provides the ability to 
Learner when minimum evidentiary thresholds for push grading to edge 
Competency demonstration/assertion of competency are achieved systems, or to be 
preserved within the 
competency framework 
for maturity 
Update The Competency Management Service shall be able to CASS 
Learner obtain verification of untrusted evidence from a trusted 
Competency authority 
Update The Competency Management Service shall ensure that | CASS May be an offline process 
Learner achievement of a credential requires review and for maturity level one and 
Competency _| approval by an authorized approval authority two 
Update The Competency Management Service shall evaluate the | CASS 
Learner trust in evidence based on the life cycle defined in the 
Competency | TLA MOM (IEEE P9274.3.1) 
Update The Competency Management Service shall update the | CASS 
Learner learner profile on learner competency states 
Competency 
Update The Competency Management Service shall generate an | CASS 
Learner “assessed” xAPI message if a test activity is completed 
Competency 
Update The Competency Management Service shall generate a | CASS* Might be separate service, 
Learner “socialized” xAPI message if an instrumented social support LEM 
Competency media post was captured and addressed a media or 
competency element 
Update The Competency Management Service shall generate a | CASS 
Learner “verified” xAPI message if an untrusted piece of evidence 
Competency is separately approved by a trusted agent 
Update The competency system shall continuously update the CASS* May be extra process 
Learner state of assigned goals 
Competency 
Skill Decay 
Skill Decay TLA compliant systems shall track the requirement for CASS LMS or HR system may 
proficiency, check ride, or continuing education units for have business rules for 
conferred credentials lower maturity levels that 
can also accomplish 
Skill Decay TLA compliant systems shall allow admins, OICS, content | CASS User controlled business 


managers, and Curriculum managers to set proficiency 
timers and content requirements to a user interest group 
or user 


logic for proficiency alerts. 
Works in concert with 
notification system 
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Provide The Competency Management Service shall allow CASS 
Configuration | authorized users to create, read, update and delete 
Control of CF_ | elements of a competency framework 
Over Time 
Provide The Competency Management Service shall generate an | CASS 
Configuration | alert when an element has been modified 
Control of CF 
Over Time 
Provide The Competency Management Service shall maintain a CASS 
Configuration | record of changes (user, authority, name-value pairs) 
Control of CF 
Over Time 
Compatibility Translation 
Compatibility | The Competency Management Service shall provide a CASS Dependent on 
Translation mechanism to allow mapping of one competency Competency Management 
framework to another Service migration at 
maturity level three 
Compatibility | The Competency Management Service shall provide a CASS Dependent on 
Translation mechanism to allow mapping of one credential Competency Management 
framework to an equivalent credential Service migration at 
maturity level three 
Compatibility | The Competency Management Service shall provide a CASS 
Translation mechanism to allow mapping of one credential 
framework to an equivalent credential with assigned 
experiences to close any gaps 
Compatibility | The Competency Management Service shall provide for | CASS Dependent on 
Translation import and export of competency framework data whole Competency Management 
or in part Service migration at 
maturity level three 
Credential Management Service 
Credential The Credential Management Service shall maintain an CASS Might include linked data 
Management | auditable log of trust and/or evidence that led to the (JSON-LD) in xAPI, can 
credential include archival of LMS 
data for maturity levels 
one and two 
Credential The Credential Management Service shall preserve a CASS May require full 
Management | digitally signed badge showing the credential achieved, competency and 
active date, conferral authority, and conferees name and credential system with 
service number blockchain or similar 
technology on top of 
JSON-LD 
Credential The Credential Management Service shall provide a CASS 
Management _| validated digital export of the digitally signed badge 
Credential The Credential Management Service shall be able to CASS May be offline process in 
Management | assign user specified business rules for validating maturity level one and two 
credentials to a user interest group (beyond 
assessments) to include source agency, military record, 
time in rate/job, assignment, multiple signature 
authorities 
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Credential The Credential Management Service shall be able to CASS The “signature obtained” 
Management | generate non-repudiable alerts to OICS role users to becomes the trigger to 
establish required conferral and validation signatures update credential 
Credential The Credential Management Service shall monitor CASS May be interface to/from 
Management | achievement of CEU/PDU requirements and issue de- AGILE at maturity level 
credentials or updates as necessary one and two, need to 
update record as well as 
create notifications 
Credential The Credential Management Service shall provide a user | CASS 
Management | configurable name for digital badges (e.g. diploma, 
certificate, badge) 
Credential TLA compliant systems shall updated the learner profile | CASS 
Management | with all completed and in progress credentials for users 
Credential TLA compliant systems shall validate credentials required | CASS Instructor permissions for 
Management | for a user acting in an OICS role for access, observation, LMS in maturity level 1 
or assessment 
Credential TLA compliant systems shall provide a secure digital CASS Portable credentials will 
Management | badge for showing a credential has been conferred require MOA with the 
parent organizations 
Credential TLA compliant systems shall provide an administrator CASS 
Management | configurable type for naming type of credential: (e.g. 
degree/diploma, badge, license, certificate, and 
professional rating) 
Credential The TLA Credential Management Service shall be able to | CASS* Tied to ELRR work, may be 
Management __| export credentials using OpenBadge3 separate process 
Credential The TLA Credential Management Service shall preserve | CASS* Tied to ELRR work, may be 
Management | the chain of evidence between globally discoverable separate process 
credentials, local copies of credentials, the assertions of 
underlying competencies, and the evidence gathered for 
the assertion. 
Credential The TLA credential chain of evidence shall be severable | CASS* Tied to ELRR work, may be 
Management | for purpose of transport or data federation (e.g. separate process 
assertions sent without evidence in message payload, 
but still preserving discoverable links) 
Credential The underlying competencies that each credential CASS* 
Management | represents will be defined using Credential Transparency 
Description Language (CTDL) and will reference the 
specific RCDs that each credential represents 
Search Functions 
Search The TLA Competency Management Service shall allow for | 2019 Stub class for testing, 
Functions searches of competency objects based on job, credential pending CASS 
or as part of an unassociated top-level competency 
Search The TLA competency search function shall return all 2019 Stub class for testing, 
Functions lower level competency definition objects from a pending CASS 
selected competency or credential 
Search The TLA competency search function shall display the 2019 Stub class for testing, 
Functions directed acyclic graph of relationships between pending CASS 
competency definition objects 
Search The TLA competency search function shall display 
Functions supporting details for selected competency graphs 
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3.2.4 User Interface 


The TLA policy framework assumes that each service group (Activity Registry and Resource 
Management, Competency Management, and Learning Event Management) will have its own organic 
user interfaces, but that a single sign-on capability enabled from a common portal will streamline user 
access to comply with the concept-of-execution for the entire TLA instance as a federation of 
components. The portal operates through redirects and filters and serves as the access point to core 
services, edge decision support services, and potentially a launch context for web-based clients of 
learning devices. 


The ADL Initiative uses the cybernetic system concept of “control loops” to model the learner as an 
entity requiring ‘location’ and ‘navigation’ over the course of their career through different learning 
experiences. The control loops are aligned with “aperture” settings in the portal to ensure that only data 
and time scales applicable to one control loop are presented at the same time, as an aid to the user. 
Especially for the higher-level control loops of 4 & 5 which may rely on access to other integrated 
systems. Navigated learning experiences include: 


e Control Loop 1: Improving a learner’s mastery of competencies within the current learning activity; 
e Control Loop 2: Optimizing a learner’s progress toward a credential; 

e Control Loop 3: Prioritizing the pursuit of credentials or activities to meet requirements for a job; 

e Control Loop 4: Planning education and training goals for an overall career trajectory; and 

e Control Loop 5: Providing options for supporting post career transition and retraining. 


A simplified version of the control loops is shown in Figure 3. Table 4 lists the detailed requirements for 


the portal. 
: Outcomes Mission 
Formal Informa ork/Life : 
Training Training xperience Effectiveness 


Control Loop 1: Optimize Current Activity 


Control Loop 2: Optimize 
Progress Towards Credential 
Credential 


Prima 
Edu enh Career/Job 
Selection 

Control Loop 3: 

Optimize Credentials Job/ /Gig 

for Current Job 
Control Loop 4: 
Optimize Career 
Career Trajectory 


Career / MOS / AFSC / NEC 


fey al ge) Mele) Mba) (-\ Wm Ot-1e-1-)  Cles-1 J 


Figure 3. TLA Control Loops. The “sensors” of the control loops are the xAPI statements generated from learning 
devices and operational data, and the “actuators” are the planning of learning events. The five control loops are 
constantly operating in parallel, but they provide a convenient way to limit and categorize data displayed in 
decision support aids, and the MOM profile serves to organize those filters. 
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Table 4. Requirements for the User Interface Portal. 


Header 


Requirement 


Decision Support Management Service 


Priority Comments 


General Requirements 


General The TLA User Interface shall provide decision support 2019 
Requirements view of the collected experience data 
General The decision support service shall enable search and 2019 
Requirements filtering of data 
General The decision support service shall enable retrieval across | 2019 
Requirements multiple transactional LRS (i.e. enterprise analytics) 
General The decision support service shall be able to reconcile 2019 
Requirements user identity across enclaves 
Instructor Review 
Instructor Review | The decision support service shall enable analysis of 
efficacy of curriculum 
Instructor Review | The decision support service shall enable analysis of 
efficacy of assessments 
Instructor Review | The decision support service shall enable an analysis of 
learner performance distribution 
Instructor Review | The decision support service shall enable achievement 
velocity analysis by user interest group for OICS 
Content Manager Review 
Content Manager |The decision support service shall enable analysis of 
Review efficacy of supporting materials 
Content Manager |The decision support service shall enable analysis of cost 
Review effectiveness of activities, content and resources 
Content Manager |The decision support service shall enable analysis of 
Review media suitability for training to a competency 
Competency Management Review 
Competency The decision support service shall enable analysis of Outyear 
Management competency frameworks suitability for assigned jobs 
Review 
Competency The decision support service shall enable analysis of Outyear 
Management effectiveness of proficiency requirements for credentials 
Review 
Competency The decision support service shall enable analysis of Outyear 
Management robustness of credentialing processes 
Review 
Personnel Manager Review 
Personnel TLA compliant systems decision support shall enable Outyear 
Manager Review _| analysis of workforce proficiency 
Personnel TLA compliant systems decision support shall enable Outyear 
Manager Review _| analysis of manning levels for projected job requirements 
Personnel TLA compliant systems decision support shall enable Outyear 
Manager Review | analysis of facility and OIC manpower efficacy 
Personnel TLA compliant systems decision support shall enable Outyear 
Manager Review _| analysis of learner velocity through training pipeline 
Personnel TLA compliant systems decision support shall enable Outyear 
Manager Review _| analysis of proficiency duty cycle 


Learner Decision Support 
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Header 


Learner Decision 
Support 


Requirement 

TLA compliant systems decision support shall enable 
individual learning progression planning for current 
class/event 


Priority Comments 


Control Loop 1 


Learner Decision 
Support 


TLA compliant systems decision support shall enable 
individual learning progression planning for current 
competency/badge/certificate/diploma goal 


Control Loop 2 


permission level and identity 


Learner Decision |TLA compliant systems decision support shall enable Outyear | Control Loop 3 
Support individual learning progression planning for next 
assignment goal 
Learner Decision |TLA compliant systems decision support shall enable Outyear | Control Loop 4 
Support individual learning progression planning through current 
career arc 
Learner Decision |TLA compliant systems decision support shall enable Outyear | Control Loop 5 
Support individual learning progression plans for service transition 
or change of career 
Common Portal 
Common Portal The portal shall provide for a user login and 2 factor 2019 
authentications 
Common Portal The portal shall employ single -sign on for all connected Keycloak 
enclaved services 
Common Portal The portal shall display an appropriate classification 2019 
Common Portal The portal shall display a consent to monitoring banner 2019 
Common Portal The portal shall allow a user to user to switch between 2019 
allowable roles 
Common Portal The portal shall require a unique login for a user to actin | 2019 
the administrator role 
Common Portal The portal shall be able to support operation when 
installed at the unclassified (NIPR), GENSER Secret (SIPR) 
and TS SCI (JWICS) levels 
Common Portal The portal shall support access to data and services at Any CDS is outside the 
lower enclaves when MLS cross domain access is TLA compliant 
provided systems enclave, and 
MLS should be always 
available in the form 
of air gapped, 
encrypted transfer 
Common Portal The portal shall enable access to TLA system resources 2019 The core portal 
applicable to user permission level functions may be 
through an organic 
interface in the initial 
maturity levels, but 
should be a common 
access point for level 
three 
Common Portal TLA system resources other than portal will only allow 2019 
access by administrators 
Common Portal The common portal shall filter all service access by user 2019 
permission level 
Common Portal The common portal shall filter all data access by user 2019 As applicable for 2019 


pending P4STLA 
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Common Portal The common portal shall allow interface with the alert 2019 
and notification system 

Common Portal The common portal shall allow a user to select decision 2019 
support dashboards 

Common Portal The common portal shall allow a user to select Learning | 2019 
Goal Management 

Common Portal Portal Learning Goal Management shall include goal 
selection and prioritization 

Common Portal Portal Learning Goal Management shall include goal and 
sub-goals path planning 

Common Portal The common portal shall allow a user to select Learning | 2019 
Task Management 

Common Portal Portal Learning Task Management shall include selection | 2019 
of pending and assigned tasks 

Common Portal Portal Learning Task Management shall include OICS 
approval of requested tasks 

Common Portal Portal Learning Task Management shall include selection 
of assigned, shared, or created content set lists 

Common Portal The common portal shall allow a user to select Learning | 2019 
Event Planning 

Common Portal Portal Learning Event Planning shall use the selected 2019 
goals if selected during goal planning 

Common Portal Portal Learning Event Planning shall use the priority goals | 2019 
by default to filter experience searches 

Common Portal Portal Learning Event Planning shall allow easy movement | 2019 
between activity centric and goal centric planning (l.e. 
without excessive user actions) 

Common Portal Portal Learning Event Planning shall include experience 
selection and scheduling 

Common Portal The common portal shall allow a user to select User 2019 Limited, includes 
Management group management in 

outyear 

Common Portal Portal User Management shall include group membership 

Common Portal Portal User Management shall include and CRUD 2019 
functions for unprotected user data 

Common Portal The common portal shall display username without 2019 Clean room only 
maintaining an association to SSO token persistently in 
the local context 

Common Portal The common portal shall display current goals, tasks, 2019 
suspense dates job, competency in work state, credential 
state, and identity group memberships 

Common Portal The common portal shall allow for minimize/maximize 2019 
functions for each class of data 

Common Portal The common portal presentation shall preserve 2019 
hierarchical relationships between goals/tasks 
jobs/credentials and competencies as applicable 

Common Portal Learning Task Management shall include scheduled 2019 
activities and recommended activities 

Common Portal The common portal shall allow a user to view the ECC 2019 
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Apertures 


performance data "apertures" that include current lesson 
progress/content, current course progress/ planning, 
planning for next credential, planning for next job, career 
trajectory planning, transition to new career 


Common Portal The common portal shall allow a user to view the entire 2019 
Experience Index 
Common Portal The common portal shall allow a user to use the portal as | 2019 
an experience client device 
Common Portal Identity and configuration settings shall pass to the client | 2019 
context without requiring reentry 
Common Portal The client interface and TLA planning interface shall exist | 2019 No launcher app 
as decoupled services 
Learning Path Apertures 
Learning Path The TLA portal shall be able to select between 2019 Loops 1 and 2 only 


Learning Path 


The TLA portal shall list milestones applicable to the 


Apertures 


career: Requirements for advancement, job alignment to 
competency, competency mapping to civilian 
competencies 


Apertures selected aperture 
Learning Path TLA compliant systems portal shall filter data for Loop 3 
Apertures progression planning for current competency/ 

badge/certificate/diploma goal (all content, progress, 

velocity, gradebook, current proficiency state 
Learning Path TLA compliant systems portal shall filter data for next Loop 4 
Apertures assignment: review of available jobs and duties along 

trajectory, projected competency state, competency gaps 
Learning Path TLA compliant systems portal shall filter data for overall Loop 5 


Alerts and Notifications 


Notifications 


Alerts and The portal shall display alerts and notifications applicable 
Notifications to the user and role logged in 
Alerts and Alerts shall require acknowledgement to clear Define alerts as 


modal, notifications as 
modeless 


Notifications 


Alerts and Notifications shall continue on a scrolling message area 
Notifications 

Alerts and The maximum retention of notifications shall be settable 
Notifications by the administrator 

Alerts and Conferral of a credential shall create an alert that learner 
Notifications is in maintenance phase 

Alerts and A Just in time training requirement inserted by a content 
Notifications manager shall create an alert 

Alerts and A regulatory or mandatory training requirement shall 
Notifications create an alert 

Alerts and An impending (~30 days) proficiency requirement shall 
Notifications create an alert 

Alerts and Changes to a previously viewed activity/content element 
Notifications shall generate a notification 

Alerts and Changes to a previously completed credential or in work 


competency/credential shall generate a notification 
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Alerts and 
Notifications 


Requirement 

Updates to user information shall create a notification to 
the user, and any OICS, or administrators with interest 
groups the user is assigned 


Priority Comments 


Alerts and 
Notifications 


Notifications from the Competency Management Service 
shall be sent to assigned learners with or working toward 
those competencies 


Notifications 


associated learner 


Alerts and OICS shall be able to send notifications to assigned user 

Notifications group learners 

Alerts and Content managers shall be able to advertise 

Notifications activities/content to sets of learners as notifications 

Alerts and Users shall be able to send notifications requesting 

Notifications mentors or tutors in topics 

Alerts and Notifications and alerts shall be able to federate across 

Notifications enclaves and agency domains 

Alerts and Learner Requested courses shall generate a notification 2019 LMS can do this now - 
Notifications to the assigned OICS may be used for demo 
Alerts and Assigned tasks shall generate a notification to the 2019 LMS can do this now - 


may be used for demo 


3.2.5 


Identity Management 


Back-end services are required for the operation of the federated cloud-based enterprise architecture 
envisioned for TLA, without directly providing learning functionality. Interface with external data sources 
and use of commercial products and protocols shall enable these capabilities. Identity management is 
concerned with non-repudiation; it allows segmentation of personally identifiable information (PII) 
across multiple, anonymized locations while still allowing for reconstruction of complete portraits of 
performance, by linking locally unique anonymization references with globally unique aliases (Universal 
Unique Identifier (UUID)) and further segregation of protected privacy information (PPI) from the UUID. 
This arrangement is depicted in Figure 4. Detailed requirements are listed in Table 5. 


Globally 
Unique ID 
Verifier 


Globally 
Unique ID 
Provider 


Privacy 
Protected 
Information 


Local Single Sign On 


Globally 
Unique ID Ses oe 
Verifier Locally 
Anonymized 
Data 


Figure 4. Segmentation of ID information. The TLA approach to managing ID leverages industry best practices to 
segment Pll, isolate PPI and use third party identity validation using open ID protocols to maintain integrity and 
non-repudiation of learner data. 
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Table 5. Detailed Requirements for Identity Management. 


Header Requirement Priority Comments 

Roles and Permissions 

Roles and TLA compliant systems shall enable login with administrator level 2019 

Permissions | privileges 

Roles and Administrator level permissions shall be able to access and modify 2019 Protection of 

Permissions | user, content, service configuration, activity, resource, and system data 
competency service data assets in test 

configuration 

Roles and Administrator level permissions shall be able to assign learners to an Registrar 

Permissions | Observer/Instructor/Controller/Supervisor (OICS) (for filtering function for 
purposes) schoolhouses 

Roles and Administrator level permissions shall be able to assign Experience 

Permissions | ownership to an OICS (for filtering purposes) 

Roles and Administrator level permissions shall be able to assign competency 

Permissions | frameworks or framework segments to a Competency Management 
Service 

Roles and Administrator level permissions shall be able to create protected user 

Permissions | identity groups with assigned users and assign access to these to 
OICS, competency, or content managers 

Roles and Administrator privileges shall include CRUD permissions by segment | 2019 

Permissions | for each of the data stores (Experience Index, LRS, Learner Profile) 

Roles and TLA compliant systems shall enable login with learner level privileges | 2019 

Permissions 

Roles and The learner access shall be able to select, deselect and prioritize goals | 2019 

Permissions | (Jobs, credentials or competencies) 

Roles and The learner access shall be able to select current scheduled courses 2019 

Permissions 

Roles and The learner access shall be able to manage (CRUD) curated experience | 2019 

Permissions | lists 

Roles and The learner access shall allow for launching of current selected 2019 

Permissions | experiences, curated lists, or assigned courses 

Roles and The learner shall be able to search the Enterprise Course Catalog 2019 

Permissions 

Roles and The learner shall be able to filter and search the entire local 2019 

Permissions | experience list 

Roles and The learner shall be able to view learner state information from the 2019 

Permissions | leaner profile 

Roles and The learner shall be able to review their personal performance data 2019 

Permissions 

Roles and TLA compliant systems shall enable login with OICS level privileges Objective for 

Permissions 2019 

Roles and OICS level permissions shall allow for logging observed practical Anomalous (non 

Permissions | exercises for assigned learners as complete- satisfactory, attempted, LMS) content 
complete-unsatisfactory captured in LRS 

in first iteration 

Roles and OICS level permissions shall allow for reviewing progress toward goal, 

Permissions | current grades and state for assigned learners 

Roles and OICS level permissions shall allow for review of assigned learner Objective for 

Permissions | performance on assigned activities 2019 

Roles and OICS level permission shall allow for review of alerts and notifications 

Permissions | sent to assigned learners 
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Roles and TLA compliant systems shall enable login with Competency 

Permissions | Management Service level privileges 

Roles and The Competency Management Service shall be able to create, read, 

Permissions | update, delete competency definition objects and relationships for 

assigned competency frameworks 
Roles and The Competency Management Service shall be able to create, read, 
Permissions | update, delete links between competency definition objects from the 
competency framework for each credential 

Roles and The Competency Management Service shall be able to create, read, 

Permissions | update, delete job, duty, gigs and competency frameworks 

Roles and TLA compliant systems shall enable login with Experience manager 

Permissions | level privileges 

Roles and The experience manager shall be able to register new activities, 

Permissions | content, or content types for a learning activity 

Roles and The experience manager shall be able to assign and attribute new 

Permissions | activities and content for learners to experience 

Roles and The experience manager shall be able to register activities and For NIPR access 

Permissions | content from within or external to the enclave to WWW or SIPR 
outside of 
enclave 

Roles and The experience manager shall be able to link content elements into Initially within 

Permissions | courses or subordinate units (phases/modules/units) Content 
Aggregation 
Model (CAM) in 
SCORM manifest 
as part of LMS, 
but refactored in 
later maturity 
levels to content 
and competency 
management 
records 

Roles and The curriculum manager shall be able to register experiences to 

Permissions | educational purpose for linked competencies 

Roles and User permission profiles shall be exportable to another federate PASTLA 

Permissions | instance of TLA compliant systems 

PPI/PII Protection/Privacy 

PPI/PII TLA compliant systems shall be able to create a locally unique 2019 

Protection/ | anonymized identity reference 

Privacy 

PPI/PII The anonymized identity token shall be used to label "user" for all 2019 

Protection/ | locally stored data 

Privacy 

PPI/PII TLA compliant systems shall otherwise use the anonymized reference | 2019 

Protection/ |when transmitting data referenced to users to another enclave 

Privacy 

PPI/PII TLA compliant systems shall send UUID to anonymization references 

Protection/ | for only requested users as a separate message from the anonymized 

Privacy performance data 
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PPI/PII UUID and anonymized reference keys shall be encrypted using FIPS 

Protection/ | 140.2 compliant encryption or higher, as appropriate to classification 

Privacy level 

PPI/PII Sensitive personal data (i.e. PPI) shall be only stored within or 2019 Scrub of xAPI 

Protection/ | transmitted from the back-end identity management service data handles 

Privacy 

PPI/PII The portal shall utilize a FIPS 140.2 approved encryption of username 

Protection/ |to be displayed when received from Identity management services 

Privacy 

PPI/PII TLA compliant systems shall employ globally unique Identities for 2019 Federated 

Protection/ | third party identification verification (UUID) Identity 

Privacy 

PPI/PII The portal shall have mechanisms to prevent human readable linkage | 2019 

Protection/ |of username and UUID 

Privacy 

PPI/PII TLA compliant systems shall be able to reconcile internal identity 2019 Use of UUID and 

Protection/ |references with UUID IRI internally 

Privacy prevent storage 
of Pll outside of 
clean room 
environment 
within enclave 

PPI/PII The portal shall only display current name when used inthe learner, | 2019 

Protection/ |admin, experience manager or Competency Management Service role 

Privacy 

PPI/PII The portal shall only display names for associated learners when used 

Protection/ |in the OICS role 

Privacy 

PPI/PII TLA compliant systems Shall be able to reconcile anonymized tokens | 2019 

Protection/ |in federated data structures (between organizations and between 

Privacy enclaves) 

PPI/PII TLA compliant systems shall enable configurable privacy settings at 

Protection/ | the individual datum value level 

Privacy 

PPI/PII TLA compliant systems shall have a mechanism to filter data exports | PS4TLA 

Protection/ | or visualization based on privacy settings 

Privacy 

Identity Groups 

Identity TLA compliant systems shall be able to associate users with identity 

Groups groups that will link data common to all users within that group 

Identity TLA identity groups shall facilitate "interest" areas for learner to 

Groups receive notifications (i.e. rather than just being pushed) 

Identity TLA identity groups shall interface with the alert and notification 

Groups system 

Identity The TLA identity groups shall be discoverable in federations for 

Groups determining applicability of learning event records 

Identity Users shall be able to create and subscribe to unprotected user 

Groups interest groups 

Identity OICS and administrators shall be able to create protected interest 2019 Class creation 

Groups groups and assign users to them 
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User Data 
User Data Identity management services shall be able to assign personal DIV-2 
attribute data 
User Data Identity management services shall be able to assign personas to a DIV-2 
user 
User Data Identity management services shall be able to assign privacy data to P4STLA 
user records 
User Data Identity management services shall be able to reconcile UUID to As applicable for 
person identity in back-end services 2019 
User Data Identity management services shall be able to reconcile identity 2019 
across enclaves (i.e. between different anonymization tokens) 
User Data Identity management services shall be able to export a user record 
audit 
User Data Identity management services shall be able to implement dynamic 
multi-factor authentication 
User Data Identity management services shall be able to resolve internal identity 
tokens to a globally unique identity 
User Data Identity management services shall integrate with privacy controls to 
prevent access to data based on locally managed policies 
User Data The access policy manager shall include local, regional and global 
business rules for data access 
User Data User data shall incorporate proper encryption/decryption for identity 
tokens and personal data 
User Data User data shall be resolvable between individuals and identity groups, 
and between multiple local identity tokens 


3.2.6 Virtualization Management 


This is a back-end service required for TLA cloud based, federated operation. Most of these services are 
provided natively in cloud-based providers like Amazon Web Services (used in the TLA Reference 
Implementation) and are at the heart of the DEVOPS method. Additional edge devices may use their 
own approaches to managing network resources, and they must comport to the rules established for 
the TLA and its native hosting environment. System-level performance requirements are also located 


here. Table 6 


lists the detailed requirements. 


Table 6. Detailed requirements for Virtualization Services. 


Header | Requirement Priority Comments 

Virtualization Management Service 

Virtualization | TLA compliant components shall utilize back-end services for 2019 

Management | dynamic endpoint management between components, data, 

Service and services. 

Virtualization | TLA compliant systems shall enable federated data services 2019 

Management| between enclaves 

Service 

Virtualization | TLA compliant systems shall leverage trusts between back-end | 2019 

Management | identity management services 

Service 

Virtualization | TLA compliant systems shall have a configuration capability 2019 

Management| that registers service and data providers that operate within 

Service the enclave, to include back-end services and data portability 
between adjacent ecoservices. 
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Virtualization | TLA compliant systems shall have a configuration capability 2019 May be manual file 
Management| that registers service and data providers that are federated to for 2019, "content 
Service the enclave, to include Simulators, digital devices, and remote in the wild" 
hosted content or learning management systems 
Virtualization | TLA compliant systems portal shall use a RESTful 2019 Intent to use Open 
Management| implementation to connect to enclave and federated data API and REST as part 
Service services of TLA interface 
spec and federation 
development 
process 
Virtualization | TLA compliant systems shall provide a registration service for 2019 Works with 
Management| all enclave and federated data sources to manage URI blocks, governance system 
Service permission holders, and path name/URL/IP for resources above to provide 
physical registry 
Virtualization | TLA compliant systems shall utilize mechanisms to dynamically | 2019 AWS back-end 
Management| track and update network and physical hosting of virtual 
Service private networks, computational resources and containers in a 
contracted Platform as a service (cloud) environment 
Virtualization | TLA compliant systems shall verify core data and services System health 
Management | (competency and learner profile, LRS/Learning event, diagnostics 
Service management, experience catalog, Competency Management 
Service, competency framework, learner profile) are available 
to conduct training session 
Virtualization | TLA compliant systems shall have sufficient load balancing, AWS back-end 
Management| failover, and redundancy to maintain Ao >98% 
Service 
Virtualization | TLA compliant systems shall have data backups to prevent loss AWS back-end 
Management | of data even in event of core data or service failure 
Service 
Virtualization | TLA compliant systems shall have sufficient memory and Server sizing 
Management| storage resources to maintain 10 years of credential trust 
Service audit trail (i.e. preservation of reviews and awards) 
Virtualization | TLA compliant systems shall have sufficient memory and Server sizing 
Management| storage resources to maintain evidentiary records for 
Service competency in accordance with local regulations 
Virtualization | TLA compliant systems shall have sufficient memory and 2019 Rough sizing for 
Management | computational power for 90% peak duty cycle for 120% of experiments as well 
Service projected user base as deployed 
systems 
Virtualization | TLA compliant systems shall have a security audit system that AWS back-end 
Management| logs server down time, VM load shifting, attempted 
Service communication time outs, unauthorized users or devices, and 
rejected xAPI statements 
Virtualization | TLA compliant systems shall implement NIST 800 controls for 
Management| identity, access, zero trust device management, behavioral 
Service controls and authentication. 
Interfaces 
Interfaces The portal shall enable single sign on for all subordinate 2019 Keycloak 
services accessed through the portal 
Interfaces TLA compliant systems shall use existing back-end services 2019 Objective 
(e.g. LDAP/Active Directory) for identity management 
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Header 


| Requirement 


Priority Comments 


Interfaces TLA compliant systems shall provide an access point to Virtual | 2019 
Private Cloud resources for client devices in the federation 

Interfaces TLA compliant systems shall comply with cybersecurity 
policies for the installed enclave 


3.2.7 Edge Devices 


TLA edge devices include all learning record providers that create learning or assessment opportunities. 
The TLA supports any edge system topology as long as the boundary is compliant with the TLA internal 
interfaces and business rules. External topologies within the ecosystem may include fully TLA compliant 
devices, server-based systems that manage their own client communications and report compliant xAPI 
from the server, or systems with their own LRS and profiles, with gateway systems on the boundary to 
convert internal data to TLA compliant data. Detailed Requirements are listed in Table 7. 


Table 7. Edge Device Requirements. 


Header | Requirement Priority Comments 

Learning Device Requirements 

Learning Device | Learning devices shall have a boundary to the TLA core 2019 

Requirements that conforms to the MOM profile for xAPI messages 

Learning Device | Learning device content packaging shall use the same For single SCO 

Requirements lowest level grain size for independent "bookmarks", content with complex 

performance reporting, and catalog registration metadata courses, or ITS or self- 

regulated devices 
where some of the 
device manages 
activity sequencing 
independent of core 

Learning Device | Learning device boundaries shall synchronize user As required for 2019 

Requirements identity with core identity management 

Learning Device | Learning devices shall be registered with the Experience 2019 

Requirements Index as activities and resources 

Learning Device | Learning devices shall comply with cybersecurity 

Requirements requirements for the local enclave 

Learning Device | "Bring Your Own" learning devices shall comply with zero In-the-wild devices 

Requirements trust risk architectures 

Learning Device | Learning devices shall be deployed as LTI Tools in Investigate for 2020 

Requirements conjunction with the launch features of the LEM 

Learning Device | Learning devices with locally installed content shall 2019 

Requirements synchronize that content identification with the 

Experience Index 

Learning Device | Learning devices shall use noisy LRS if they generate xAPI | 2019 

Requirements that does not conform to the MOM profile 

Learning Device | Learning devices with their own profile shall register that 

Requirements profile IAW IEEE P9274.2.1 

Learning Device | Learning device boundaries shall register endpoints to As required for 2019 

Requirements transactional LRS as part of device registration 

Learning Device | All performance adjudication shall be performed outside 2019 


Requirements 


of the core boundary 


Learning Device 
Requirements 


Learning devices that perform fine grained performance 
adaptation shall be registered at the highest level and 
report lower level results as required 


Investigating 
competency object 
"ownership" 
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3.3 External Interface Requirements 


The TLA ecosystem relies on several DoD level data repositories to work and can leverage non-education 
and training databases to provide additional capabilities for education and training. 


3.3.1. Enterprise Learner Record Repository (ELRR) Registry 


Each instance (i.e. “enclave”) of TLA compliant computational assets will have its own learner profile 
with learner records. Organizations will have equities in their own data but will lack visibility to users 
outside of their own purview. Learners will move from organization to organization throughout a career. 
The approach taken with the ELRR Registry is to keep authoritative data with the owners of the systems 
where these data are generated. Therefore, not every record associated with a learner is in the same 
database. This requires a federated approach to managing trusted, authoritative data where this 
information is pulled as needed from the authoritative data stores. 


A central mechanism to aggregate and reconcile these disparate data is needed to provide a complete 
portrait of learner performance. Currently, all records are copied and centralized, with error correction 
being an infrequent manual process. The TLA architecture conceives of a centralized registry that lists 
locations for all users where any records would be located, enabling discovery, but where the actual 
records would be served up from the individual providers. Service requests will come as encrypted 
Representational State Transfer (REST) calls using the UUID for each learner. 


3.3.2. Enterprise Course Catalog (ECC) Registry 


Like the ELRR registry, multiple sites may have activity and content registered as an available course. Not 
every site will own course content, but actual assignments of course ownership (e.g. curriculum control 
managers) will probably change slowly. The ECC registry aids in the discovery of all locations that report 
course content -- to include integration with existing systems such as the Catalog of Naval Training 
Courses (CANTRAC). 


3.3.3, DoD Schema Control 


The JavaScript Object Notation (JSON) provides for the use of linked data, which must trace back to the 
schema.org repository. Schema.org is not globally accessible from .mil or protected network assets, so a 
DoD unique repository is required at each security enclave to replicate the functionality. Schema control 
and linked data is essential to preserving the chains of custody and evidence without performance 
degradation that comes from retransmitting the entire list every time. Linked data provides for data 
integrity, and disaster resiliency, and the schema repository anchors the linked data strategy 


3.3.4 Universal Unique Identification (UUID) 


The UUID provides a globally unique way of referencing all DoD personnel without resorting to PPI such 
as name, social security number, etc. The UUID and third-party logins provide a federated way to 
acknowledge user identity as they move through the ecosystem, while still preserving the local 
anonymization protocols for PII segmentation. The UUID are foundational to non-repudiation and data 
integrity to provide enterprise-level data analytics and complete portraits of performance. For the DoD, 
a potential unclassified enclave UUID is the Electronic Data Interchange Personal Identifier (EDIPI) or 
Common Access Card (CAC) token that traces back to the Defense Enrollment Eligibility Reporting 
System (DEERS), an authoritative data source for DoD personnel identity. The UUID is used to support 
Federated Identity, Credential, and Access Management (FICAM). 
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Participants 


At the time this Standard was completed, the JavaScript Object Notation (JSON) Binding of Experience API (xAPI) 
Data for the Total Learning Architecture (TLA) Working Group had the following membership: 


TBD, Chair 


Jerry Gordon, Andy Johnson, and Florian Tolk, Technical Editors 


The following individual members of the balloting committee voted on this Standard. Balloters may have voted for approval, 
disapproval, or abstention. 


Also included are the following nonvoting IEEE-SA Standards Board liaisons: 


Introduction 


(This introduction is not part of IEEE Std 9274.3.2, JavaScript Object Notation (JSON) Binding of Experience API (xAPI) 
Data for the Total Learning Architecture (TLA). 


This Standard defines a set of controlled vocabulary and processes to be followed when the IEEE Standard xAPI (9274.1.1) 
is used in an environment categorized as conformant to the Total Learning Architecture. It provides a JSON binding of such 
data that is also conformant to the IEEE xAPI Profile Standard (9274.3.1). This standard defines the structure and con- 
straints of JSON data in this environment. 


The purpose of this Standard is to allow the creation of semantically interoperable instances of learning data services ex- 
changing xAPI across learning environments that adopt the TLA. This Standard uses a JSON encoding that is also conform- 
ant to the xAPI and xAPI Profile standards, which allows for interoperability and the exchange of xAPI data between all 
components of the TLA. 
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IEEE Standard for Learning Technology— 
JavaScript Object Notation (JSON) Binding of Experi- 
ence API (xAPI) Data for the Total Learning Architec- 
ture (TLA) 


1.0 Overview 


The scope and purpose of this Standard are discussed in sections 1.1 and 1.2. 


1.1 Scope 


This Standard defines a set of controlled vocabulary and processes to be followed when the IEEE Standard xAPI (9274.1.1) 
is used in an environment categorized as conformant to the Total Learning Architecture. It provides a JSON binding of such 
data that is also conformant to the IEEE xAPI Profile Standard (9274.3.1). This standard defines the structure and con- 
straints of JSON data in this environment. This standard may be used as an xAPI profile in that elements of this standard 
may be extracted and used in other profiles, or independent of a TLA-conformant technology. 


1.2 Purpose 


The purpose of this Standard is to allow the creation of interoperable xAPI instances across learning environments that 
adopt the TLA. This Standard uses a JSON encoding that is also conformant to the xAPI and xAPI Profile standards, which 
allows for interoperability and the exchange of xAPI data between all components of the TLA. 


1.3 Word usage 


The word shall indicates mandatory requirements strictly to be followed to conform to the standard and from which no devi- 
ation is permitted (shall equals is required to).)? 
The word should indicates that among several possibilities one is recommended as particularly suitable, without mentioning 


or excluding others; or that a certain course of action is preferred but not necessarily required (should equals is recom- 
mended that). 


The word may is used to indicate a course of action permissible within the limits of the standard (may equals is permitted 
to). 


The word can is used for statements of possibility and capability, whether material, physical, or causal (can equals is able 
to). 


2.0 Normative references 


The following referenced documents are indispensable for the application of this Standard. For dated references, only the 
edition cited applies. For undated references, the latest edition of the referenced document (including any amendments or 
corrigenda) applies. 


' The use of the word must is deprecated and shall not be used when stating mandatory requirements, must is used only to describe unavoidable 
situations. 


> The use of will is deprecated and shall not be used when stating mandatory requirements, will is only used in statements of fact. 
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IEEE Std 9274.1.1, Experience API (xAPI) Standard.? 

IEEE Std 9274.3.1, Experience API (xAPI) Profile Standard.* 

Cmi5 Specification > 

RFC 8256: The JavaScript Object Notation (JSON) Data Interchange Format® 


3.0 Definitions, acronyms, and abbreviations 


Definitions and acronyms are defined in sections 3.1 and 3.2, respectively. 


3.1 Definitions 


For the purposes of this Standard, the following terms and definitions apply. The Authoritative Dictionary of IEEE Stand- 
ards Terms [B7]’ should be referenced for terms not defined in this clause. 


Actor: An individual, organization, technology, or other provider of data within a learning experience or acting within the 
TLA. In an xAPI statement, the “doer” of the statement. 


Authoritative LRS: A classification of LRS that stores a record of learner proficiency that is aligned with an individual’s 
associated competency frameworks. It has the purpose of disallowing certain persons access to certain types of data. Only 
an Observer, Instructor, Controller, Supervisor (OICS) can access an authoritative LRS. The Authoritative LRS stores digi- 
tally signed xAPI records of conferred credentials and other competency assertions. 


Competency: Formally defined, organized, and structured description of knowledge, skills, attributes, and other (KSAOs) 
characteristics that can be used to manage human capital. Each competency can have a wide range of associated metadata 
(e.g. description, type, scope, level, and context) and associated resources (e.g. assessments, operations manuals, and train- 
ing content). 


Competency Framework: A data model for describing, referencing, and sharing competency definitions, primarily in the 
context of learning and development. It provides a formal representation of the key characteristics of a competency, inde- 
pendently of its use in any particular context. It enables interoperability among learning systems that deal with competency 
information by providing a means for them to refer to common definitions with common meanings. 


Experience API (xAPI): An IEEE Standard (9274.1.1) that establishes data formats and protocols for learning experience 
data. Most of the requirements around the creation, storage, and retrieval of JSON data. 


JavaScript Object Notation (JSON): A format of JavaScript that has specific structure and properties. These include use 
of structured name/value pairs and an ordered list of values. 


Learning Record Provider (LRP): A system/service that creates xAPI data and sends it to an LRS. An LRP is responsible 
for the quality and structure of xAPI data. 


Learning Record Store (LRS): The basic storage and retrieval web service, often implemented as a system, for xAPI data. 


Master Object Model (MOM): The TLA policy framework includes a Master Object Model (TLA MOM) that specifies 
event triggers between TLA core components. The TLA MOM is an xAPI profile that defines the activity streams that cre- 
ate the events to manage the TLA federation execution. 


Noisy LRS: An LRS without additional data restrictions seen in an Authoritative LRS. This LRS is typically associated 
with a specific learning activity, device, or system and is used to house all xAPI statements generated by that system. These 


3 IEEE publications are available from the Institute of Electrical and Electronics Engineers, Inc., 445 Hoes Lane, Piscataway, NJ 08854, USA 
(http://standards.ieee.org/). 


4 IEEE publications are available from the Institute of Electrical and Electronics Engineers, Inc., 445 Hoes Lane, Piscataway, NJ 08854, USA 
(http://standards.ieee.org/). 


° https://github.com/AICC/CMI-5_Spec_Current/blob/quartz/cemi5_spec.md 


° https://tools.ietf.org/html/rfc8259 
7 The numbers in brackets correspond to those of the bibliography in Annex A. 
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statements are used to drive additional interventions or functionality within that system but only the TLA MOM verbs are 
communicated to other TLA components or activities. 


Observer/Instructor/Controller/Supervisor (OICS): An actor with increased permission to access learner data. The only 
allowable user of an authoritative LRS. 


Profile: Additional rules to be applied from a base specification. This document is a profile of xAPI, meaning that to be 
conformant to the profile, the technology shall also be conformant to xAPI. 


Statement: A basic unit of learning experience data as defined in the xAPI standard. A statement is formatted using JSON 
and has among its properties, actor, verb, and object. 


Total Learning Architecture (TLA): A research and development activity sponsored by the ADL Initiative and conducted 
in collaboration with stakeholders from across the defense community, professional standards organizations, and commer- 
cial industry. The TLA project will result in a collection of policy, specifications (including this document), and standards 
for connecting “any device, anywhere, any time” to generate learning-related data enabling next-generation learning that is 
integrated, personalized, and data-driven. 


Transactional LRS: An LRS without additional data restrictions seen in an authoritative LRS. While the noisy LRS col- 
lects all xAPI statements, the transactional LRS only collects learner data that is conformant to the TLA Master Object 
Model (MOM). This allows learner performance information to be normalized as it is processed by a Competency Manage- 
ment System. A transactional LRS is trusted within TLA. 


Verb: The most defining property in an xAPI statement — the “action” of the statement. A verb is an IRI, often shortened 
when being described, that uniquely identifies a tracked interaction. 


3.2 Acronyms and abbreviations 
The following acronyms and abbreviations are commonly used in this document. 


JSON — JavaScript Object Notation 

LRP: Learning Record Provider 

LRS: Learning Record Store 

OICS: Observer/Instructor/Controller/Supervisor 
TLA: Total Learning Architecture 

xAPI: Experience API 


4.0 Conformance 


Conformance to this Standard is discussed in sections 4.1 through 4.5. Please see Section 1.3, Word Usage, to determine the 
nature of requirements found in this section. The TLA proscribes multiple levels of conformance. There are requirements for 
conformance that are both in-scope and out of the scope of this document. The requirements for xAPI data are listed in sections 
4.1 through 4.5. It is expected that all TLA systems, such as LRS and LRP, follow the requirements appropriate to their 
function in a TLA environment. 


4.1 TLA conformance level 1 
An instance of TLA level 1... 


Shall conform to the xAPI Standard 

Shall conform to the xAPI Profile Standard 

Should conform to the cmi5 Specification (regarding xAPI statements) 

Shall implement general requirements as found in section 5.1 

Shall implement statements using the “certified” verb with all requirements fulfilled from section 5.1.36 

Should implement statements using the “completed” verb. If implemented, shall be done so with all requirements 
fulfilled from section 5.1.3 


Copyright © 2020 IEEE. All rights reserved. - This is an unapproved IEEE Standards Draft, subject to change. 


2019 Total Learning Architecture Report (November 2019) — Appendix B — Draft MOM Specification | B-3 


Should implement statements using the “passed” verb. If implemented, shall be done so with all requirements ful- 
filled from section 5.1.4 

Should implement statements using the “failed” verb. If implemented, shall be done so with all requirements ful- 
filled from section 5.1.5 

Shall implement statements using the “success” result in at least one of the statements that use the “completed,” 
“passed,” or “failed” verbs 

May implement statements with verbs found in section 5.1. If implemented, shall be done so with all requirements 
fulfilled from the corresponding section in 5.1 

May implement statements with verbs not found in this specification 

Shall send statements to their appropriate LRS as defined in section 5.2 

Should generate statements in appropriate systems as defined in section 5.3 


4.2 TLA conformance level 2 
An instance of TLA level 2... 
e Shall follow requirements of all TLA level 1 
e Shall conform to the cmi5 Specification (regarding xAPI statements) 
e Shall implement statements using the “completed” verb with all requirements fulfilled from section 5.1.3 
e Shall implement statements using the “passed” verb with all requirements fulfilled from section 5.1.4 
e — Shall implement statements using the “passed” verb using the “success” result property if the object type is “as- 
sessment” 
e Shall implement statements using the “failed” verb with all requirements fulfilled from section 5.1.5 
e Shall implement statements using the “failed” verb using the “success” result property if the object type is “‘assess- 
ment” 
e Shall implement statements using the “launched” verb with all requirements fulfilled from section 5.1.2 
e Shall implement statements using the “initialized” verb with all requirements fulfilled from section 5.1.9 
e Shall implement statements using the “waived” verb with all requirements fulfilled from section 5.1.1 
e Shall implement statements using the “satisfied” verb with all requirements fulfilled from section 5.1.6 
e Shall implement statements using the “abandoned” verb with all requirements fulfilled from section 5.1.7 
e Shall implement statements using the “terminated” verb with all requirements fulfilled from section 5.1.8 
4.3. TLA conformance level 3 
An instance of TLA level 3... 
e Shall follow requirements of all TLA level 1-2 
e Shall implement authority in xAPI statements that use verbs that require an authoritative source, as defined in sec- 
tion 5.1 
e Shall implement statements using the “assessed” verb with all requirements fulfilled from section 5.1.26 
e Shall implement statements using the “contextualized” verb with all requirements fulfilled from section 5.1.27 
e Shall implement statements using the “located” verb with all requirements fulfilled from section 5.1.28 
e Shall implement statements using the “socialized” verb with all requirements fulfilled from section 5.1.29 
e Shall implement statements using the “captured” verb with all requirements fulfilled from section 5.1.30 
e Shall implement statements using the “asserted” verb with all requirements fulfilled from section 5.1.31 
e Shall implement statements using the “validated” verb with all requirements fulfilled from section 5.1.32 
e Shall implement statements using the “inferred” verb with all requirements fulfilled from section 5.1.33 
e Shall implement statements using the “qualified” verb with all requirements fulfilled from section 5.1.34 
e Shall implement statements using the “certified” verb with all requirements fulfilled from section 5.1.35 
e Shall implement statements using the “verified” verb with all requirements fulfilled from section 5.1.36 
e Shall implement statements using the “conferred” verb with all requirements fulfilled from section 5.1.37 
4.4 TLA conformance level 4 
An instance of TLA level 4... 
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Shall follow requirements of all TLA level 1-3 

Shall implement statements using the “organized” verb with all requirements fulfilled from section 5.1.11 
Shall implement statements using the “prioritized” verb with all requirements fulfilled from section 5.1.12 
Shall implement statements using the “curated” verb with all requirements fulfilled from section 5.1.13 
Shall implement statements using the “projected” verb with all requirements fulfilled from section 5.1.14 
Shall implement statements using the “recommended” verb with all requirements fulfilled from section 5.1.10 
Shall implement statements using the “planned” verb with all requirements fulfilled from section 5.1.15 
Shall implement statements using the “requested” verb with all requirements fulfilled from section 5.1.16 
Shall implement statements using the “approved” verb with all requirements fulfilled from section 5.1.17 
Shall implement statements using the “augmented” verb with all requirements fulfilled from section 5.1.18 
Shall implement statements using the “explored” verb with all requirements fulfilled from section 5.1.19 
Shall implement statements using the “clarified” verb with all requirements fulfilled from section 5.1.20 
Shall implement statements using the “directed” verb with all requirements fulfilled from section 5.1.21 
Shall implement statements using the “scheduled” verb with all requirements fulfilled from section 5.1.22 
Shall implement statements using the “evaluated” verb with all requirements fulfilled from section 5.1.23 
Shall implement statements using the “tracked” verb with all requirements fulfilled from section 5.1.24 
Shall implement statements using the “surveyed” verb with all requirements fulfilled from section 5.1.25 


TLA conformance level 5 


An instance of TLA level 5... 


5.0 


Shall follow requirements of all TLA level 1-4 

Shall implement statements using the “recruited” verb with all requirements fulfilled from section 5.1.38 
Shall implement statements using the “appraised” verb with all requirements fulfilled from section 5.1.39 
Shall implement statements using the “detailed” verb with all requirements fulfilled from section 5.1.40 
Shall implement statements using the “mobilized” verb with all requirements fulfilled from section 5.1.41 
Shall implement statements using the “employed” verb with all requirements fulfilled from section 5.1.42 
Shall implement statements using the “schooled” verb with all requirements fulfilled from section 5.1.43 
Shall implement statements using the “promoted” verb with all requirements fulfilled from section 5.1.44 
Shall implement statements using the “screened” verb with all requirements fulfilled from section 5.1.45 
Shall implement statements using the “selected” verb with all requirements fulfilled from section 5.1.46 
Shall implement statements using the “reclassified” verb with all requirements fulfilled from section 5.1.47 
Shall implement statements using the “released” verb with all requirements fulfilled from section 5.1.48 
Shall implement statements using the “restricted” verb with all requirements fulfilled from section 5.1.49 


TLA xAPI JSON-binding definition 


ATLA solution is “learner-centric.” This means all learner activity should be retrievable from any LRS (e.g., even those 
with minimally conformant behavior) through the querying mechanism in the xAPI standard. This standard describes the 
reporting portion of a TLA solution, which decentralizes state management within the systems. Most of this management is 
done through the use of xAPI statements. Any TLA data reporting that is not xAPI is beyond the scope of this specification. 
Sections 5.1-5.3 describe the data requirements of xAPI statements within the TLA. A comprehensive description of the use 


of TLA data can be found in Annex E. 


5.1 Statement data requirements 


This section describes requirements for xAPI statements within the TLA. An xAPI statement has an actor, verb, and object, 
and other properties. A statement is from an authoritative source if and only if the authority property is traceable back to the 


statement provider. 


e  xAPI statements shall be conformant to the xAPI Standard 


Copyright © 2020 IEEE. All rights reserved. - This is an unapproved IEEE Standards Draft, subject to change. 


2019 Total Learning Architecture Report (November 2019) — Appendix B — Draft MOM Specification 


e xAPI statements using verbs in this document shall implement properties as described (see Note below) in the 
templates in sections 5.1.1-5.1.49 

e xAPI statements using verbs in this document may implement properties that are not represented in the templates 
in sections 5.1.1-5.1.49 

e xAPI statements shall be constructed such that the learner is represented by the actor, object, a specific context 
extension as represented by the templates in sections 5.1.1-5.1.49 

e = The actor in any xAPI statement shall be of object type “Agent” 

e The actor in any xAPI statement shall contain an “account, even if the actor is not the learner” 


Each verb in an xAPI statement determines which template to follow, corresponding to a specific piece of learning evidence 
that is tracked. The naming convention of these verbs indicates an active model, where the actor is the “doer” of the verb. 
The learner is an integral part of every xAPI statement. When an xAPI statement is sent directly as a result of the learner’s 
action, such as a user requesting to take a course, the actor of that statement is the learner. When something is happening to 
the learner as a result of another actor, such as an instructor or system, the learner is represented in a context extension, such 
that the activity may remain in the object portion of the statement. The rationale for this design is to minimize the number of 
necessary extensions within this profile, instead of relying on Activity types that the LRS can keep definitions of more eas- 
ily. All other necessary requirements for xAPI statement properties, in addition to those in the xAPI Standard, are listed in 
the xAPI Profile templates listed in sections 5.1.1-5.1.49. Among these properties are objects with type “activity,” which 
often represent the learning construct of the statement. More information about each type of object can be found in Annex 
D. 


Note: Any property listed in a template with a specific value (including booleans) shall retain that value in a statement cre- 
ated that follows the template. In properties where there is not a specific value, these templates contain in capital letters re- 
quirements that shall be followed. These requirements are not “shall” “should” or “shall not” but should be translated as 
such. The use of “RECOMMENDED?” in the template indicates a “should” requirement. The use of “EXCLUDED” in the 
template indicates a “shall not” requirement. The use of “REQUIRED” in the template indicates a “shall” requirement. 
These requirements may be accompanied by an explanatory text and/or other requirements on the data. The use of brackets 
indicates a choice of one or more of the elements within the bracket. 


5.1.1 Waived 
Verb: 
id: “https://w3id.org/xapi/adl/verbs/waived”, 


display: “waived”, 


definition: “Indicates that the learning activity requirements were met by means other than completing the 
activity. A waived statement indicates the actor may skip the activity.” 


Object: 
id: “”’, 
objectT ype: Activity 
definition: 
type: 
["https://w3id.org/xapi/tla/activity-types/activity”, 
"https://w3id.org/xapi/tla/activity-types/assessment”, 
"https://w3id.org/xapi/tla/activity-types/competency"] 


5.1.2 Launched 

Verb: 
id: “http://adInet.gov/expapi/verbs/launched”, 
display: “launched”, 


B-6 | 2019 Total Learning Architecture Report (November 2019) — Appendix B — Draft MOM Specification 


P9274.3.2_D1_June2020 


definition: “Indicates the user started a service. This does not always need to be a specific activity but can be 
a service provider as well.” 


Object: 
id: “’, 
objectT ype: Activity 
definition: 
type: 

["https://w3id.org/xapi/tla/activity-types/activity”, 
"https://w3id.org/xapi/tla/activity-types/assessment”, 
"https://w3id.org/xapi/tla/activity-types/competency"| 

Context: 


Context Activities: EXCLUDED 


5.1.3 Completed 
Verb: 

id: “http://adlnet.gov/expapi/verbs/completed”, 

display: “completed”, 

definition: “Indicates the actor finished or concluded the activity normally” 
Object: 

id: “’, 

objectType: Activity 

definition: 

type: 
["https://w3id.org/xapi/tla/activity-types/activity”, 

"https://w3id.org/xapi/tla/activity-types/assessment" ] 
Result: 

Success: RECOMMENDED 

Duration: RECOMMENDED 


5.1.4 Passed 


Verb: 

id: “http://adlnet.gov/expapi/verbs/passed”, 

display: “passed”, 

definition: “Indicates the actor completed an activity to the standard” 
Object: 


4, 6099 
id: “’, 
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Result: 


5.1.5 
Verb: 


Object: 


Result: 


5.1.6 
Verb: 


Object: 


B-8 | 


objectT ype: Activity 
definition: 


type: “https://w3id.org/xapi/tla/activity-types/assessment" 


Score: 

Scaled: RECOMMENDED 
Success: TRUE 
Completion: TRUE 


Failed 


id: “http://adlnet.gov/expapi/verbs/failed”’, 
display: “failed”, 


definition: “Indicates the actor did not complete an activity to standard” 


id: 6699 Z 
objectT ype: Activity 
definition: 


type: “https://w3id.org/xapi/tla/activity-types/assessment" 


Score: 

Scaled: RECOMMENDED 
Success: FALSE 
Completion: TRUE 


Satisfied 


id: “https://w3id.org/xapi/adl/verbs/satisfied”, 
display: “satisfied”, 


definition: “Indicates that the authority or activity provider determined the actor has fulfilled the criteria of 
the object or activity by means other than completing the activity” 


id: “’, 
objectType: Activity 
definition: 
type: 
["https://w3id.org/xapi/tla/activity-types/activity”, 
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"https://w3id.org/xapi/tla/activity-types/assessment" ] 


5.1.7. Abandoned 

Verb: 
id: “https://w3id.org/xapi/adl/verbs/abandoned”, 
display: “abandoned”, 


definition: “Indicates that the activity or session was abnormally terminated by a learner's action (or due to a 
system failure)” 


Object: 
id: “’, 
objectT ype: Activity 
definition: 
type: 
["https://w3id.org/xapi/tla/activity-types/activity”, 
"https://w3id.org/xapi/tla/activity-types/assessment" ] 
Result: 
Duration: RECOMMENDED 


The duration property should, at a minimum, be set as the total session time, calculated as the time 
between the 'launched' statement and the last statement (of any kind) issued by the exercise. Imple- 
menters should also use other (software specific) methods (if available) to determine if the total ses- 
sion time was longer. 


Completed: FALSE 


5.1.8 Terminated 
Verb: 

id: “http://adlnet.gov/expapi/verbs/terminated”’, 

display: “terminated”, 

definition: “Indicates the actor has completed their session normally” 
Object: 

id: “’, 

objectType: Activity 

definition: 

type: 
["https://w3id.org/xapi/tla/activity-types/activity”, 

"https://w3id.org/xapi/tla/activity-types/assessment" ] 
Result: 

Duration: RECOMMENDED 
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The duration property should, at a minimum, be set as the total session time, calculated as the time 
between the 'launched' statement and the last statement (of any kind) issued by the exercise. Imple- 
menters should also use other (software-specific) methods (if available) to determine if the total ses- 


sion time was longer. 


Completed: FALSE 


5.1.9 Initialized 
Verb: 
id: “http://adInet.gov/expapi/verbs/initialized”’, 
display: “initialized”, 
definition: “Indicates that the activity was started.” 
Object: 
id: “’, 
objectType: Activity 
definition: 
type: 
["https://w3id.org/xapi/tla/activity-types/activity”, 
"https://w3id.org/xapi/tla/activity-types/assessment" ] 
Result: EXCLUDED 


5.1.10 Recommended 
Verb: 
id: “http://w3id.org/xapi/tla/verbs/recommended”’, 


display: “recommended”, 


definition: “Indicates the learner was given the recommendation to follow a career path, work toward a 
learning objective, or perform a learning activity by the actor” 


Object: 
id: “’, 
objectT ype: REQUIRED 
definition: 
type: 
["https://w3id.org/xapi/tla/activity-types/activity_cluster’’, 
"https://w3id.org/xapi/tla/activity-types/competency”, 
"https://w3id.org/xapi/tla/activity-types/assessment”, 
"https://w3id.org/xapi/tla/activity-types/activity”’, 
"https://w3id.org/xapi/tla/activity-types/career”’, 
"https://w3id.org/xapi/tla/activity-types/badge”, 
"https://w3id.org/xapi/tla/activity-types/job"| 
Result: EXCLUDED 
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Context: 
Extensions: 
https://w3id.org/xapi/tla/extensions/learner REQUIRED 


shall be the learner to whom this data applies to 


5.1.11 Prioritized 


Verb: 
id: “http://w3id.org/xapi/tla/verbs/prioritized”, 
display: “prioritized”, 
definition: “Indicates the actor filtered goals associated with select content, usually listing what competen- 
cies are demonstrated in recently viewed content” 
Object: 
id: “’, 
definition: 


type: 
["https://w3id.org/xapi/tla/activity-types/competency”, 
"https://w3id.org/xapi/tla/activity-types/career’”’, 
"https://w3id.org/xapi/tla/activity-types/badge”, 
"https://w3id.org/xapi/tla/activity-types/job"] 
Result: EXCLUDED 
Context: 
Extensions: 


https://w3id.org/xapi/tla/extensions/evidence: REQUIRED 


This will be an array of activities that were used in the query. 


5.1.12 Organized 
Verb: 

id: “http://w3id.org/xapi/tla/verbs/organized”, 

display: “organized”, 

definition: “Indicates the actor filtered content that aligns to specific goal” 
Object: 

id: *’, 

objectT ype: REQUIRED 

definition: 

type: 
["https://w3id.org/xapi/tla/activity-types/activity_cluster’’, 
"https://w3id.org/xapi/tla/activity-types/assessment”’, 
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"https://w3id.org/xapi/tla/activity-types/activity"] 
Result: EXCLUDED 
Context: 

Extensions: 


https://w3id.org/xapi/tla/extensions/evidence: REQUIRED 


This will be an array of the viewed or completed Competencies 


5.1.13 Curated 

Verb: 
id: “http://w3id.org/xapi/tla/verbs/curated”’, 
display: “curated”, 


definition: “Indicates the actor was presented a list of activity recommendations over time, based on selected 
goal with recursive depth, and learner preferences, what set of content will achieve mastery in the 
ordered sub-goals” 


Object: 
id: “’, 
definition: 
type: RECOMMENDED 
Result: 
Extensions: 
https://w3id.org/xapi/tla/extensions/recommendation_order: REQUIRED 
This is an array of just activity recommendations statement references, in the order they 
were provided 
5.1.14 Projected 
Verb: 
id: “http://w3id.org/xapi/tla/verbs/projected”’, 
display: “projected”, 


definition: “Indicates the actor was presented a list of goal recommendations over time, based on selected 
goal with recursive depth, what set of content can achieve mastery in the ordered sub-goals” 


Object: 
id: “’, 
definition: 
type: RECOMMENDED 
Result: 
Extensions: 
https://w3id.org/xapi/tla/extensions/recommendation_order: REQUIRED 


This is an array of just activity recommendations statement references in the order they 
were provided 
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5.1.15 Planned 


Verb: 

id: “http://w3id.org/xapi/tla/verbs/planned”’, 

display: “planned”, 

definition: “Indicates that the actor assigned themselves a new learning goal, without needing approval” 
Object: 

id: “’, 

definition: 


type: 
["https://w3id.org/xapi/tla/activity-types/competency”, 
"https://w3id.org/xapi/tla/activity-types/career’’, 
"https://w3id.org/xapi/tla/activity-types/badge”, 
"https://w3id.org/xapi/tla/activity-types/job"] 
Result: EXCLUDED 


5.1.16 Requested 

Verb: 
id: “https://w3id.org/xapi/adb/verbs/requested”’, 
display: “requested”, 


definition: “Indicates the actor needed or demanded an object or another actor. Requested indicates a com- 
ment that is shared with peers as a group or Coach as a trainer. The request to coach or help prompts 
users to respond by giving them coaching credit.” 


Object: 
id: “’, 
definition: 
type: 
["https://w3id.org/xapi/tla/activity-types/activity_cluster’’, 
"https://w3id.org/xapi/tla/activity-types/assessment”, 
"https://w3id.org/xapi/tla/activity-types/activity"] 
Result: EXCLUDED 


5.1.17 Approved 
Verb: 
id: “http://w3id.org/xapi/tla/verbs/approved”’, 
display: “approved”, 
definition: “Indicates an OICS approved a requested activity for the given learner.” 


Object: 
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id: “”, 
SHOULD be the ID of the “Requested” Statement being approved 
definition: 
type: 
["https://w3id.org/xapi/tla/activity-types/activity_cluster’’, 
"https://w3id.org/xapi/tla/activity-types/assessment”, 
"https://w3id.org/xapi/tla/activity-types/activity"] 
Result: EXCLUDED 


Context: 
Extensions: 
https://w3id.org/xapi/tla/extensions/learner REQUIRED 


shall be the learner to whom this data applies to 


5.1.18 Augmented 


Verb: 
id: “http://w3id.org/xapi/tla/verbs/augmented”, 
display: “augmented”, 
definition: “Indicates the actor searched content on an active learning goal, viewing what other 
goals/branches can be related based on an active goal tree” 
Object: 
id2“"5 
definition: 


type: 
["https://w3id.org/xapi/tla/activity-types/activity_cluster’’, 
"https://w3id.org/xapi/tla/activity-types/assessment”, 
"https://w3id.org/xapi/tla/activity-types/activity"] 
Result: EXCLUDED 
Context 
Extensions 
https://w3id.org/xapi/tla/extensions/evidence: REQUIRED 


This should be a resolvable identifier to the learning goal(s) used for this augmented event, 
but is open-ended to allow future mechanisms 


5.1.19 Explored 


Verb: 
id: “http://w3id.org/xapi/tla/verbs/explored”, 
display: “explored”, 
definition: “Indicates the actor searched active learning goals related to specific content, viewing what 
other content may trigger related goals, based on the active goal and recently completed content” 
Object: 
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id: “°’, 
definition: 
type: 
["https://w3id.org/xapi/tla/activity-types/competency”, 
"https://w3id.org/xapi/tla/activity-types/career’’, 
"https://w3id.org/xapi/tla/activity-types/badge”, 
"https://w3id.org/xapi/tla/activity-types/job"] 
Result: EXCLUDED 
Context: 
Extensions: 
https://w3id.org/xap1/tla/extensions/evidence: REQUIRED 


It should be a resolvable identifier to the learning goal(s) used for this explored event but is 
open-ended to allow future mechanisms 


5.1.20 Clarified 


Verb: 
id: “http://w3id.org/xapi/tla/verbs/clarified”, 
display: “clarified”, 
definition: “Indicates the actor queried what other content may also reinforce the current learning goal, af- 
ter completing content” 
Object: 
id: *’, 
definition: 


type: 
["https://w3id.org/xapi/tla/activity-types/competency”, 
"https://w3id.org/xapi/tla/activity-types/career”’, 
"https://w3id.org/xapi/tla/activity-types/badge”, 
"https://w3id.org/xapi/tla/activity-types/job"] 
Result: EXCLUDED 
Context: 
Extensions: 
https://w3id.org/xap1/tla/extensions/evidence: REQUIRED 


It should be a resolvable identifier to the content and goal(s) used for this clarified event 
but is open-ended to allow future mechanisms 


5.1.21 Directed 
Verb: 
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id: “http://w3id.org/xapi/tla/verbs/directed”, 
display: “directed”, 


definition: “Indicates the actor was assigned a learning goal by another party” 


Object: 
id: “’, 
definition: 
type: 
["https://w3id.org/xapi/tla/activity-types/competency”, 
"https://w3id.org/xapi/tla/activity-types/career”’, 
"https://w3id.org/xapi/tla/activity-types/badge”, 
"https://w3id.org/xapi/tla/activity-types/job"| 
Result: EXCLUDED 
Context: 
Extensions: 
https://w3id.org/xapi/tla/extensions/learner REQUIRED 


It shall be the learner to whom this data applies 


5.1.22 Scheduled 


Verb: 

id: “http://w3id.org/xapi/tla/verbs/scheduled”’, 

display: “scheduled”, 

definition: “Indicates the actor scheduled an activity or lesson” 
Object: 

1d: 

definition: 


type: 
["https://w3id.org/xapi/tla/activity-types/activity_cluster’’, 
"https://w3id.org/xapi/tla/activity-types/assessment”, 
"https://w3id.org/xapi/tla/activity-types/activity"] 
Result: EXCLUDED 
Context: 


Extensions: 
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https://w3id.org/xapi/tla/extensions/due_date: REQUIRED® https://w3id.org/xapi/tla/exten- 
sions/learner REQUIRED 


shall be the learner to whom this data applies to 


5.1.23 Evaluated 


Verb: 

id: “http://w3id.org/xapi/tla/verbs/evaluated”, 

display: “evaluated”, 

definition: “Indicates the learner(s) appeared in a Measure of Effectiveness (MOE) search” 
Object 

id: “’, 

definition: 


type: 
["https://w3id.org/xapi/tla/activity-types/activity_cluster’’, 
"https://w3id.org/xapi/tla/activity-types/assessment”, 
"https://w3id.org/xapi/tla/activity-types/activity"] 
Result: EXCLUDED 


Context: 
Extensions: 


https://w3id.org/xapi/tla/extensions/learner REQUIRED 


shall be the learner to whom this data applies to 


5.1.24 Tracked 


Verb: 

id: “http://w3id.org/xapi/tla/verbs/tracked”, 

display: “tracked”, 

definition: “Indicates the learner(s) appeared in a competency search” 
Object: 

id: “’, 

definition: 


type: "https://w3id.org/xapi/tla/activity-types/competency" 
Result: EXCLUDED 


Context: 


8 Prior to IEEE Standardization of xAPI, the following requirement existed: This shall be in the same time zone and for- 
mat as the rest of the timestamps in this statement 
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Extensions: 
https://w3id.org/xapi/tla/extensions/learner REQUIRED 


shall be the learner to whom this data applies to 


5.1.25 Surveyed 

Verb: 
id: “http://w3id.org/xapi/tla/verbs/surveyed”, 
display: “surveyed”, 


definition: “Indicates the learner(s) appeared in a Measure of Performance (MOP) search” 


id: *°’, 
shall point to a node in the Competency Management System with type of MOP 
definition: 
type: 
["https://w3id.org/xapi/tla/activity-types/assessment”’, 
"https://w3id.org/xapi/tla/activity-types/activity”, 
"https://w3id.org/xapi/tla/activity-types/career”’, 
"https://w3id.org/xapi/tla/activity-types/badge”, 
"https://w3id.org/xapi/tla/activity-types/job"| 
Result: EXCLUDED 


Context: 
Extensions: 


https://w3id.org/xapi/tla/extensions/learner REQUIRED 


shall be the learner to whom this data applies to 


5.1.26 Assessed 


Verb: 
id: “http://w3id.org/xapi/tla/verbs/assessed” 
display: “assessed”, 
definition: “Indicates the actor completed assessments in a way that will cause a change in their authorita- 
tive learner state” 
Object: 
id: *’, 
definition: 
type: "https://w3id.org/xapi/tla/activity-types/competency" 
Result: 


Duration: RECOMMENDED 
Completed: EXCLUDED 


Score: 
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Scaled: RECOMMENDED 
Success: REQUIRED 
Context: 
Extensions: 
https://w3id.org/xapi/tla/extensions/evidence: REQUIRED 
shall be a resolvable identifier to the xAPI statement(s) that resulted in this statement 
https://w3id.org/xapi/tla/extensions/confidence: REQUIRED 


shall be a number between -1 and 1 displaying how likely the learner is to have mastered 
the assessed competency, -1 being sure they have not, and I being certain they have 


5.1.27 Contextualized 

Verb: 
id: “http://w3id.org/xapi/tla/verbs/contextualized” 
display: “contextualized”, 


definition: “Indicates the user performed several connected learning activities that should result in a change 
in their authoritative learner state” 


Object: 
id: *’, 
definition: 
type: https://w3id.org/xapi/tla/activity-types/competency 
Context: 
Extensions: 
https://w3id.org/xapi/tla/extensions/evidence: REQUIRED 
shall be a resolvable identifier to the xAPI statement(s) that resulted in this statement 
https://w3id.org/xapi/tla/extensions/confidence: REQUIRED 


shall be a number between -1 and 1 displaying how likely the learner is to have mastered the as- 
sessed competency, -1 being sure they have not, and I being certain they have 


5.1.28 Located 


Verb: 
id: “http://w3id.org/xapi/tla/verbs/located” 
display: “located”’, 
definition: “Indicates the actor's competency state needs to be updated based on completed content changes 
in the Competency Framework” 
Object: 
1d2o 
definition: 
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type: https://w3id.org/xapi/tla/activity-types/competency 
Result: EXCLUDED 
Context: 
Extensions: 
https://w3id.org/xapi/tla/extensions/evidence: REQUIRED 
shall be a resolvable identifier to the xAPI statement(s) that resulted in this statement 
https://w3id.org/xapi/tla/extensions/confidence: REQUIRED 


shall be a number between -1 and 1 displaying how likely the learner is to have mastered the lo- 
cated competency, -1 being sure they have not, and I being certain they have 


5.1.29 Socialized 


Verb: 

id: “http://w3id.org/xapi/tla/verbs/socialized”’, 

display: “socialized”, 

definition: “Indicates the learner interacted with \"Wild\" (unscheduled) content in a social environment” 
Object 

id: “’, 

definition: 


type: 
["https://w3id.org/xapi/tla/activity-types/activity_cluster’’, 
"https://w3id.org/xapi/tla/activity-types/assessment”, 
"https://w3id.org/xapi/tla/activity-types/activity"] 
Result: EXCLUDED 


5.1.30 Captured 


Verb: 

id: “http://w3id.org/xapi/tla/verbs/captured”, 

display: “captured”, 

definition: “Indicates the learner interacted with \"Wild\" (unscheduled) content in a social environment” 
Object 

id: “’, 

definition: 


type: 
["https://w3id.org/xapi/tla/activity-types/activity_cluster’’, 
"https://w3id.org/xapi/tla/activity-types/assessment”, 
"https://w3id.org/xapi/tla/activity-types/activity"] 
Result: EXCLUDED 
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5.1.31 Asserted 


Verb: 
id: “http://w3id.org/xapi/tla/verbs/asserted” 
display: “asserted”, 
definition: “Indicates the learner has provided sufficient evidence to update the learner's measure of compe- 
tence in a given competency” 
Object: 
ide, 
definition: 


type: https://w3id.org/xapi/tla/activity-types/competency 
Result: EXCLUDED 
Context: 
Extensions: 
https://w3id.org/xapi/tla/extensions/evidence: REQUIRED 
shall be a resolvable identifier to the xAPI statement(s) that resulted in this statement 
https://w3id.org/xap1/tla/extensions/confidence: REQUIRED 


shall be a number between -1 and 1 displaying how likely the learner is to have mastered the com- 
petency, -1 being sure they have not, and 1 being certain they have 


5.1.32 Validated 


Verb: 
id: “http://w3id.org/xapi/tla/verbs/validated” 
display: “validated”, 
definition: “Indicates an OICS approved a change to a competency framework within the TLA that will af- 
fect the learners’ states” 
Object: 
id: “’, 
definition: 


type: https://w3id.org/xapi/tla/activity-types/competency 
Result: EXCLUDED 
Context: 
Extensions: 
https://w3id.org/xapi/tla/extensions/learner REQUIRED 
shall be the learner to whom this data applies to 
https://w3id.org/xapi/tla/extensions/evidence: REQUIRED 


shall be a resolvable identifier to the xAPI asserted statement was just validated 
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https://w3id.org/xapi/tla/extensions/confidence: REQUIRED 


shall be a number between -1 and 1 displaying how likely the learner is to have mastered the com- 
petency, -1 being sure they have not, and I being certain they have 


5.1.33 Inferred 


Verb: 
id: “http://w3id.org/xapi/tla/verbs/inferred” 
display: “inferred”, 
definition: “Indicates an authoritative source changed a learner's competency assertions based on a 
valid competency framework change” 
Object: 
id: °%, 
definition: 


type: https://w3id.org/xapi/tla/activity-types/competency 
Result: EXCLUDED 
Context: 


Extensions: 
https://w3id.org/xapi/tla/extensions/learner REQUIRED 


shall be the learner to whom this data applies to 
https://w3id.org/xapi/tla/extensions/evidence: REQUIRED 
shall be a resolvable identifier to the xAPI statement(s) that resulted in this statement 
https://w3id.org/xapi/tla/extensions/confidence: REQUIRED 
shall be a number between -1 and 1 displaying how likely the learner is to have mastered the com- 


petency, -1 being sure they have not, and I being certain they have 


5.1.34 Qualified 


Verb: 
id: “http://w3id.org/xapi/tla/verbs/qualified” 
display: “qualified”, 
definition: “Indicates the learner meets all the requirements for a badge, but hasn't been awarded the badge 
yet” 
Object: 
id: “’, 
definition: 


type: https://w3id.org/xapi/tla/activity-types/badge 
Result: EXCLUDED 
Context: 
https://w3id.org/xapi/tla/extensions/evidence: REQUIRED 


shall be a resolvable identifier to the xAPI statement(s) that resulted in this statement 
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5.1.35 Certified 


Verb: 
id: “http://w3id.org/xapi/tla/verbs/certified” 
display: “certified”’, 
definition: “Indicates the learner received an accreditation by an authoritative source to perform a given job 
or task” 
Object: 
ide, 
definition: 


type: https://w3id.org/xapi/tla/activity-types/job 
Result: EXCLUDED 
Context: 
Extensions: 
https://w3id.org/xapi/tla/extensions/evidence: REQUIRED 


shall be a resolvable identifier to the xAPI statement(s) that resulted in this statement 


5.1.36 Verified 


Verb: 
id: “http://w3id.org/xapi/tla/verbs/verified” 
display: “verified”, 
definition: “Indicates the authoritative source verified evidence of learning from a non-authoritative source 
as reliable data” 
Object: 
id: “’, 
definition: 


type: https://w3id.org/xapi/tla/activity-types/competency 
Result: EXCLUDED 
Context: 
Extensions: 
https://w3id.org/xapi/tla/extensions/evidence: REQUIRED 
shall be a resolvable identifier to the xAPI statement(s) that resulted in this statement 
https://w3id.org/xapi/tla/extensions/confidence: REQUIRED 


shall be a number between -1 and 1 displaying how likely the learner is to have mastered the com- 
petency, -1 being sure they have not, and I being certain they have 


https://w3id.org/xapi/tla/extensions/learner REQUIRED 


shall be the learner to whom this data applies to 
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5.1.37 Conferred 


Verb: 

id: “http://w3id.org/xapi/tla/verbs/conferred” 

display: “conferred”, 

definition: “Indicates the OICS conferred a badge to the learner in the learner context extension” 
Object: 

id: “’, 

definition: 


type: https://w3id.org/xapi/tla/activity-types/badge 
Result: EXCLUDED 
Context: 
Extensions: 
https://w3id.org/xapi/tla/extensions/evidence: REQUIRED 
shall be a resolvable identifier to the xAPI statement(s) that resulted in this statement 
https://w3id.org/xapi/tla/extensions/confidence: REQUIRED 


shall be a number between -1 and 1 displaying how likely the learner is to have mastered the com- 
petency, -1 being sure they have not, and I being certain they have 


https://w3id.org/xapi/tla/extensions/learner REQUIRED 


shall be the learner to whom this statement applies to 


5.1.38 Recruited 


Verb: 

id: “http://w3id.org/xapi/tla/verbs/recruited” 

display: “recruited”, 

definition: “Indicates the actor recruited the learner to join the ecosystem” 
Object: 

id: “’, 

definition: 


type: https://w3id.org/xapi/tla/activity-types/career 
Result: EXCLUDED 
Context: 
Extensions: 
https://w3id.org/xapi/tla/extensions/learner REQUIRED 


shall be the learner to whom this data applies to 


5.1.39 Appraised 
Verb: 
id: “http://w3id.org/xapi/tla/verbs/appraised” 
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display: “appraised”, 
definition: “OICS indicates the learner met entry criteria for jobs and assigned a career trajectory” 
Object: 
id: “’, 
definition: 
type: https://w3id.org/xapi/tla/activity-types/career 


Result: EXCLUDED 


5.1.40 Detailed 


Verb: 

id: “http://w3id.org/xapi/tla/verbs/detailed” 

display: “detailed”’, 

definition: “OICS detailed the learner to a specific job” 
Object: 

id: “’, 

definition: 


type: https://w3id.org/xapi/tla/activity-types/career 
Result: EXCLUDED 
Context 
Extensions 
https://w3id.org/xap1/tla/extensions/location: RECOMMENDED 
shall be the physical location the learner has been detailed to 
https://w3id.org/xap1/tla/extensions/permanent_change_ of station: RECOMMENDED 
shall be a boolean marking if this is a PCS or a different detail event 
https://w3id.org/xapi/tla/extensions/unit_identification_code: RECOMMENDED 
shall be a unique code for the learner's unit 
https://w3id.org/xapi/tla/extensions/learner REQUIRED 


shall be the learner to whom this data applies to 


5.1.41 Mobilized 


Verb: 

id: “http://w3id.org/xapi/tla/verbs/mobilized” 

display: “mobilized”, 

definition: “OICS mobilized the learner to a state of on duty” 
Object: 


4, 6699 
id: “’, 
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definition: 
type: https://w3id.org/xapi/tla/activity-types/career_state 
Result: EXCLUDED 
Context 
Extensions 
https://w3id.org/xapi/tla/extensions/location: RECOMMENDED 
shall be the physical location the learner has been mobilized to 
https://w3id.org/xapi/tla/extensions/unit_identification_code: RECOMMENDED 
shall be a unique code for the learner's unit 
https://w3id.org/xapi/tla/extensions/learner REQUIRED 


shall be the learner to whom this data applies to 


5.1.42 Employed 


Verb: 

id: “http://w3id.org/xapi/tla/verbs/employed” 

display: “employed”, 

definition: “OICS employs the actor such that they started work doing their job” 
Object: 

id: °°, 

definition: 


type: https://w3id.org/xapi/tla/activity-types/career_state 
Result: EXCLUDED 
Context 
Extensions 
https://w3id.org/xap1/tla/extensions/location: RECOMMENDED 
shall be The physical location the learner has been employed at 
https://w3id.org/xap1/tla/extensions/unit_identification_code: RECOMMENDED 
shall be a unique code for the learner's unit 
https://w3id.org/xapi/tla/extensions/learner REQUIRED 


shall be the learner to whom this data applies to 


5.1.43 Schooled 
Verb: 
id: “http://w3id.org/xapi/tla/verbs/schooled” 
display: “schooled”, 
definition: “OICS has enrolled the learner in a schooling system” 
Object: 
id: “°’, 
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definition: 
type: 
["https://w3id.org/xapi/tla/activity-types/competency”, 
"https://w3id.org/xapi/tla/activity-types/career’’, 
"https://w3id.org/xapi/tla/activity-types/badge”, 
"https://w3id.org/xapi/tla/activity-types/job"] 
Result: EXCLUDED 
Context 
Extensions 
https://w3id.org/xap1/tla/extensions/location: RECOMMENDED 
shall be the physical location the learner has been employed at 
https://w3id.org/xap1/tla/extensions/unit_identification_code: RECOMMENDED 
shall be a unique code for the learner's unit 
https://w3id.org/xapi/tla/extensions/learner REQUIRED 


shall be the learner to whom this data applies to 


5.1.44 Promoted 
Verb: 
id: “http://w3id.org/xapi/tla/verbs/promoted” 
display: “promoted”, 
definition: “OICS has changed a learner’s rank, either up or down” 
Object: 
id: “’, 
definition: 
type: "https://w3id.org/xapi/tla/activity-types/rank" 
Result: EXCLUDED 
Context: 
Extensions: 
https://w3id.org/xapi/tla/extensions/learner REQUIRED 


shall be the learner to whom this data applies to 


5.1.45 Screened 
Verb: 
id: “http://w3id.org/xapi/tla/verbs/screened” 


display: “screened”, 
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definition: “OICS screened learner for a potentially narrower career trajectory, and passed through a \"gate\" 
within their career trajectory” 


Object: 
id: “’, 
definition: 
type: https://w3id.org/xapi/tla/activity-types/career 
Result: EXCLUDED 
Context 
Extensions 
https://w3id.org/xapi/tla/extensions/evidence: RECOMMENDED 


The reason the learner screened for a career path should be a resolvable identifier to xAPI 
data 


https://w3id.org/xap1/tla/extensions/epiration: RECOMMENDED 
shall be a timestamp of the time the screening expires 
https://w3id.org/xapi/tla/extensions/learner REQUIRED 


shall be the learner to whom this data applies to 


5.1.46 Selected 


Verb: 
id: “http://w3id.org/xapi/tla/verbs/selected” 
display: “selected”, 
definition: “OICS selected learner based on criteria for a potentially wider career trajectory, opening up new 
career possibilities” 
Object: 
1d27, 
definition: 


type: https://w3id.org/xapi/tla/activity-types/career 
Result: EXCLUDED 
Context 
Extensions 
https://w3id.org/xapi/tla/extensions/evidence: RECOMMENDED 


The reason the learner was selected for a career path should be a resolvable identifier to 
xAPI data 


https://w3id.org/xap1/tla/extensions/epiration: RECOMMENDED 
shall be a timestamp of the time the screening expires 
https://w3id.org/xapi/tla/extensions/learner REQUIRED 


shall be the learner to whom this data applies to 
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5.1.47 Reclassified 


Verb: 
id: “http://w3id.org/xapi/tla/verbs/reclassified” 
display: “reclassified”, 
definition: “Indicates the actor changed career paths, putting them on a completely different or brand 
new career trajectory” 
Object: 
id: “’, 
definition: 


type: https://w3id.org/xapi/tla/activity-types/career_state 
Result: EXCLUDED 


5.1.48 Released 


Verb: 

id: “http://w3id.org/xapi/tla/verbs/released”’ 

display: “released”, 

definition: “Indicates OICS released the learner from the learning environment” 
Object: 

id: “°”, 

definition: 


type: https://w3id.org/xapi/tla/activity-types/career_state 
Result: EXCLUDED 


Context: 
Extensions: 
https://w3id.org/xap1/tla/extensions/reason: REQUIRED 
shall be text/String that describes the reason the learner has left the learning environ- 
ment 
Context: 


https://w3id.org/xapi/tla/extensions/learner REQUIRED 


shall be the learner to whom this data applies to 


5.1.49 Restricted 

Verb: 
id: “http://w3id.org/xapi/tla/verbs/restricted” 
display: “restricted”, 


definition: “Indicates OICS temporarily restricted the learner from some (possibly all) participation within 
the learning environment” 
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Object: 
id: *’, 
definition: 
type: https://w3id.org/xapi/tla/activity-types/career_state 
Result: EXCLUDED 
Context: 
Extensions: 
https://w3id.org/xapi/tla/extensions/restriction: REQUIRED 
shall be text/String that describes the reason the learner has been restricted 
https://w3id.org/xap1/tla/extensions/reason: REQUIRED 


shall be the timestamp corresponding to when the restriction is lifted. May be NULL(e.g. if 
the restriction will not expire) 


https://w3id.org/xapi/tla/extensions/learner REQUIRED 


shall be the learner to whom this data applies to 


5.2 LRS storage requirements 


Within a TLA environment, there are different levels of trust and access for different types of LRSs. The level of trust, user 
roles within those systems, and transfer of data between LRSs are out of scope of this document. Brief definitions of the 
types of LRSs can be found in the section 3.1. 


e = The following requirements exist for all LRSs within a TLA environment: 
e All LRSs shall conform to the xAPI Standard 


e All LRSs may use the voided verb as found in the xAPI Standard. This requirement supercedes other requirements 
below. 


e Noisy LRSs may allow statements that use any verb 
e Transactional LRSs should allow all statements with verbs in section 5.1 that authoritative LRSs do not allow 


o Note: depending on the conformance level, recommended practices change. For example, at conform- 
ance levels | and 2, there is no expected authoritative LRS, so all data is stored in the transactional LRS. 


e Transactional LRSs shall not allow statements that use any verb not in section 5.1. 


e Authoritative LRSs shall not allow statements that use any verb except for the following verbs in section 5.1: 


> qualified > conferred > certified 
> validated > inferred > asserted 
> verified > recruited > appraised 
> detailed > mobilized > employed 
> schooled > screened > selected 
> promoted > reclassified > released 


e Authoritative LRSs should allow statements that use any verb in the list above. 
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5.3 TLA environment reporting requirements 


A TLA compliant learning environment is composed of various functional groups. Each TLA functional group serves a dif- 
ferent role with different requirements for reporting data (sending statements to an LRS). The functional groups may exist in 
any number of configurations of software and hardware, as part of distinct from devices used as “learning activities” to conduct 
learning. It is for this reason that the following are only recommended practices, as defining what a system is or is not simply 
by what is reported is not sufficient. 


For a description of each of the TLA systems, please refer to Annex E. 


Only a competency management system shall send statements with the “certified” verb, unless the TLA environment is level 
1 or 2. 


5.3.1 Competency management function 


A level | or 2 TLA environment does not have a competency management system; rather other undefined systems shall send 
the required statements. 


A competency management system should send statements with the following verbs, as appropriate, and in accordance with 
the TLA conformance level as defined in section 4.1-4.5: 


> surveyed > evaluated > tracked 
> located > assessed > asserted 
> validated > inferred > qualified 
> verified > conferred > certified 


A competency management system should not send statements that do not contain the verbs stated in the previous require- 
ment. 


An OICS may send statements on behalf of a competency management system. 
5.3.2 Learning event management function 


A learning event management system should send statements with the following verbs, as appropriate, and in accordance with 
the TLA conformance level as defined in sections 4.1-4.5: 


> launched > waived > — satisfied 
> abandoned > recommended > organized 
> curated > requested > approved 
> augmented > clarified > directed 
Si | Broauaed > projected > planned 
> explored > captured 
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A learning event management system should not send statements that do not contain the verbs stated in the previous require- 
ment. Learning event management may be performed by multiple systems, by manual interfaces, wholly or in part by at- 
tached learning devices as activity providers that have contextual or adaptive features, or any combination thereof. The 
learning event management function may exist as part of, in conjunction with, or in addition to the mechanism used to cata- 
log representative learning experiences (e.g. an activity index, content catalog, or experience index). 


An OICS may send statements on behalf of a learning experience event system. 


5.3.3 Activity provider function 


Activity providers are the devices that generate the learning records (i.e. LRP) in response to learners conducting learning 
events. Traditional Learning Management Systems (LMS) serving Shareable Courseware Object Reference Model 
(SCORM) compliant courseware is an example of an activity provider, although simulators, observation tools, and any 
number of modern mobile learning devices may also be activity providers. Advanced providers like intelligent tutoring sys- 
tems may include activity provider functions as well as some learning event management functions. 


An activity provider should send statements with the following verbs, as appropriate, and in accordance with the TLA con- 
formance level as defined in sections 4.1-4.5: 


> completed > passed > failed 


> satisfied > terminated > initialized 
An activity provider should not send statements that do not contain the verbs stated in the previous requirement. 


An OICS may send statements on behalf of an activity provider. 


5.3.4 Human capital management function 


A human capital management system should send statements with the following verbs, as appropriate, and in accordance 
with the TLA conformance level as defined in sections 4. 1-4.5: 


> launched > waived > satisfied 
> abandoned > recommended > organized 
> curated > requested > approved 
> augmented > clarified > directed 
> scheduled 


A human capital management system should not send statements that do not contain the verbs stated in the previous require- 
ment. 


An OICS may send statements on behalf of a human capital management system. 
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Annex A 


(informative) 
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Annex B 


(informative) 


Intended use of verbs 


A learner-centric view of all verbs. Verbs here may appear to be passive to provide context to the intended effect on the 
learner but will be proper (active) xAPI statements when implemented. The term experience refers to a combination of 
learning activities (the electronic device or format that is used to conduct the learning -e.g. a kindle) and content (the file or 
resource that is experienced, (e.g. an electronic publication) 


TLA Level 1: 


Certified: Indicates the learner received an accreditation by an authoritative source (OICS) to perform a given job 
or task. 


Completed: Indicates the learner finished or concluded an experience normally. Should include the success result 
field. 


Passed: Indicates the learner completed an experience to standard. Used in assessment. 
Failed: Indicates the learner failed to complete an experience to standard. Used in assessment. 


Initialized: Indicates that the learner successfully loaded an experience. This is different from “launched”, as it 
relates to a specific experience (such as a chapter in a book instead of just opening the book). 


TLA Level 2: 


Launched: Indicates the learner started a service. This does not always need to be a specific experience but can be 
a service provider as well. 


Waived: Indicates that the learning experience requirements were met by means other than completing the experi- 
ence. A waived statement is used to indicate that the experience may be skipped by the learner. 


Satisfied: Indicates that the authority or experience provider determined the learner has fulfilled the criteria of the 
object or experience by means other than completing the experience. 


Abandoned: Indicates that the AU session was abnormally terminated by a learner's action (or due to a system 
failure). 


Terminated: Indicates the leamer has completed their session normally. 


Planned: Indicates that the learner assigned themselves a new learning goal, without needing approval from an 
OICS. 


Requested: Indicates the learner needed or demanded an object or another OICS or learner. Requested indicates a 
comment that is shared with peers as a group or a coach as a trainer. The request for coaching or help prompts us- 
ers to respond giving them coaching credit. Can also include a request to take a class or do a course. 


Directed: Indicates the learner was assigned a learning goal by an OICS. 


Approved: Indicates an OICS approved for a requested experience for the given learner. 


TLA Level 3: 


B-34 | 


Assessed: Indicates the learner completed assessments in a way that will cause a change in their authorita- 
tive learner state. 


Contextualized: Indicates the learner performed several connected learning activities that should result in a 
change in their authoritative learner state. 


Located: Indicates the learner's competency state needs to be updated based on completed experiences and 
changes in the Competency Framework. 
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Socialized: Indicates the learner interacted with "wild" (unscheduled) experience in a social environment. 
Captured: Indicates the learner interacted with "wild" (out of network) or unplanned experience. 


Asserted: Indicates the learner has provided sufficient evidence to update the learner's measure of competence in a 
given competency. 

Validated: Indicates an OICS approved a change to a competency framework within the TLA that will affect 
learners’ states. 

Inferred: Indicates an OICS changed a learner's competency assertions based on a valid competency framework 
change. 

Qualified: Indicates the learner meets all the requirements for a badge but hasn't been awarded the badge yet. 


Conferred: Indicates the learner was given a badge by an OICS. 


Verified: Indicates the learner had evidence of learning from a non-authoritative source verified as reliable data by 
an authoritative source. 


TLA Level 4: 


Organized: Indicates the leamer filtered experience that aligns to a specific goal. 


Prioritized: Indicates the learner filtered goals associated with select experience, usually listing what competen- 
cies are demonstrated in recently viewed experience. 


Curated: Indicates the learner was presented a list of experience recommendations over time, based on a se- 
lected goal with recursive depth, and on learner preferences, what set of experience will achieve mastery in the 
ordered sub-goals. 


Projected: Indicates the learner was presented a list of goal recommendations over time, based on a selected goal 
with recursive depth, what set of experience can achieve mastery in the ordered sub-goals. 


Recommended: Indicates the learner was given the recommendation to follow a career path, work towards a 
learning objective, or perform a leaming experience by the actor. 


Augmented: Indicates the learner searched for experiences on an active learning goal, viewing what other 
goals/branches can be related based on an active goal tree. 


Explored: Indicates the learner searched active learning goals related to specific experience, viewing what other 
experience may trigger related goals, based on active goal and recently completed experience. 


Clarified: Indicates the learner queried what other experience may also reinforce the current learning goal, after 
completing experience. 


Scheduled: Indicates the learner scheduled an experience or lesson. 
Evaluated: Indicates the learner(s) appeared in a Measure of Effectiveness (MOE) search. 
Tracked: Indicates the learner(s) appeared in a competency search. 


Surveyed: Indicates the learner(s) appeared in a Measure of Performance (MOP) search. 


TLA Level 5: 


Recruited: Indicates an OICS recruited the learner to join the ecosystem. 

Appraised: Indicates the learner met entry criteria for jobs and was assigned (by an OICS) a career trajectory. 
Detailed: Indicates the learner detailed to a specific job. 

Mobilized: Indicates the learner mobilized or deployed (i.e. data will be time late) on duty. 

Employed: Indicates OICS employs the learner such that they started work doing their job. 
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e Schooled: Indicates OICS has enrolled the learner in a schooling system. 
e Promoted: Indicates the OICS has changed a learner’s rank, either up or down. 


e Screened: Indicates the OICS screened learner as passed through a "gate" within their career trajectory to open up 
restricted opportunities. 


e Selected: Indicates the learner met the criteria for a potentially wider career trajectory, opening new career possi- 
bilities. 

e Transition: Indicates the learner changed career paths, putting them on a completely different and brand-new ca- 
reer trajectory. 


e Released: Indicates OICS released the learner from the learning environment. 


e Restricted: Indicates OICS temporarily restricted the learner from some (possibly all) participation within the 
learning environment. 
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Annex C 


(informative) 


Expected data flow for formal and informal learning 


Within TLA enabled systems, there is an expected flow of learner state as depicted in Figure 1. The flow may in- 
clude a deliberate or casual configuration of the learner’s environment. In deliberate learning, the sequence begins 
with the learner setting goals, planning (or being assigned plans) tasks to achieve these goals, scheduling learning 
events, and then launching learning exercises. Each is represented by a verb within this specification, and the data 
generated by learning exercise is then stored in the transactional LRS describing the order and context under which 
the learner, or the learner’s mentor (OICS) configured their own learning environment. The relationship between the 
goals, tasks, events, records of completion, evaluated assertions and conferral chain provides a discoverable audit 
trail or trusted chain of evidence. 


Point of Entry for Deliberate Learning 


4... 


Goal Planning Scheduling Launching, 


Setting 


Learning 
Device 


f Point of Entry for 
Informal Learning 
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Capturing Contextualizing Evaluating Locating 
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Bere = S=-S—=! REST Calls ------ > 
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Figure 1: Data Flow for Formal and Informal Learning. 


Once the learner completes the exercise, the newly generated data is contextualized and a trusted source (OICS or 
trusted system) takes this new information and generates a trusted record of the learner state using verification, con- 
ferral, qualification and assertion statements. These statements are stored in an authoritative LRS, that only OICSs 
and trusted systems have access to. 


This standard uses a learner-centric vocabulary, which enables the entire TLA compliant environment to run as a 
stateless system of systems. This means that the learner could interface with TLA data at any stage in the expected 
object life cycle, and the services comprising the TLA instance can execute without knowing previous or follow on 
learner states. This arrangement not only allows for a true ecosystem, since the origin of the statement doesn’t mat- 
ter, it is resilient to configuration changes with devices or services being added or subtracted over time. 
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Annex D 


(informative) 


Object types 


Object types, also known as activity types (as the object is usually of type “‘activity”), as found in the statement templates in 
sections 5.1.1-5.1.49, are used to populate the “object” field of an xAPI statement. Activity types are the various types of 
learning constructs present in the TLA. A summary of each activity type can be found below in Table 1. 


Table 1. Object Types. 
ETT Identifier 
Activity https://w3id.org/xapi/tla/activ- 


Definition 


Any generic activity an actor can interact with, that is not an assess- 


ity-types/activity ment 
Assessment https://w3id.org/xapi/tla/activ- | Any generic exercise that formally assesses the user's level of com- 
ity-types/assessment petence 
Competency https://w3id.org/xapy/tla/activ- | A collection of knowledge, skills, abilities, or other behaviors 
ity-types/competency needed to perform in a given task 
Activity Cluster | https://w3id.org/xapi/tla/activ- | Any generic collection of activities and/or assessment activities 
ity-types/activity_cluster 
Career https://w3id.org/xapi/tla/activ- | An outline of jobs and their competency and credential require- 
ity-types/career ments in a career, usually used to outline the expected learning path 
as series of verifiable milestones. 
Badge https://w3id.org/xapi/tla/activ- | An online badge that is earned after achieving multiple related com- 
ity-types/badge petencies. In this sense badge is any kind of credential, a badge, a 
certificate, a degree, a license, etc. 
Job https://w3id.org/xapi/tla/activ- | A formal job, duty, legal obligation, permanent or temporary em- 
ity-types/job ployment that requires a learner to possess some set of competen- 
cies and/or credentials 
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Annex E 


(informative) 


Description of TLA data and systems 


The learner object life cycle is divided into 3 “levels” of data or perspectives about the learner’s learning path: Learner Ca- 
reer states (LC states), depicted in Figure 2; Learner Activity states (LA states), depicted in Figure 3; and Learner Record 
Provider states (LRP states), depicted in Figure 4. All verbs are divided into Authoritative Data (depicted below in green), 
Evidence statements (depicted below in blue), and Activity Planning (depicted below in red). 


The TLA has many functional requirements for each of the component systems within a TLA environment. Those require- 
ments are beyond the scope of this document, as are some of the components in their entirety. The following limited expla- 
nations provide a description of how the system functions within a TLA environment from the standpoint of learning 
experience data. 


Competency Management System — is used to CRUD the concept and relationships that define jobs, credentials and com- 
petencies, as part of overarching competency frameworks, and calculates based on trust and evidence, the impact of learning 
events on the competencies and credentials of learners 


Learning Event Management System — is used to track the completion of the leamer data flow. Leamming events are con- 
ducted as a sequence of launching and completing an experience. The conduct of a learning event exists in context, how- 
ever, with the preparation and analysis of the learning environment, and includes goal setting, planning, scheduling, 
capturing, contextualizing, evaluating and locating exercises. 


Activity Provider — is a creator of xAPI data (a type of LRP in xAPI) that generates a learning event (statement) on its own 
(on-the-job activity) or provides the context (reader, simulator) under which learning content is experienced. 


Learning Experience — the combination of learning activities (context for the experience) and content (the resources for the 
experience) aligned for some educational purpose. 


Learner State — the hierarchy of learning goals and sequenced subgoals, the tasks defined to satisfy those goals and the 
events that address the tasks. These link to the assertions of competency and conferral of credentials for which they provide 
a discoverable audit trail of evidence. 


Human Capital Management System — is an end to end technical system used to track the recruiting, accession training, 
detailing, training, certification, promotions, screening and selections of capable manpower according to personnel jobsite 
definitions throughout the enterprise and over the entire career of personnel. 
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DoDAF ALL Views (AV) 


The Department of Defense (DoD) Architectural Framework (DoDAF) is a modeling convention used to 
convey the multilayered elements of complex systems of systems. The DoDAF mitigates the complexity 
of the ADL Initiative’s Total Learning Architecture (TLA) project for specific stakeholders by creating 
“views” that describe how requirements are implemented, and the impacts or benefits afforded to each 
stakeholder. Table 1 shows the different views and their purpose. 


Table 1. List of DoDAF Views Included With the TLA. 


View | Description Purpose 
AV-1 Overview and Summary | Describes the overall mission, vision, and resources. 
AV-2 Integrated Dictionary Describes the terms currently in use in education and training, as well as 
new TLA terms. 
Cv-1 Capability Vision Alignment of the end-state TLA policy framework to lines of effort, current 
and projected projects to inform those lines of effort, and potential 
transition jump-off points. 
CV-2 Capability Taxonomy Three-level hierarchical decomposition of capabilities enabled through TLA. 
CV-3 Capability Phasing The TLA capability maturity model used to scope future migration efforts. 
CV-6 Capability to Cross references the TLA use cases required to support each capability. 
Operational Activities Cells marked with an ‘x’ require the feature, those marked with an ‘o’ are 
Mapping enhanced by the feature. 

CV-7 Capability to Services Cross references the technical services required to support each capability. 
Mapping 

DIV-1 Conceptual Data Model | The list of governance structures and organizational information exchanges 
defining the TLA ecology. 

DIV-2 Logical Data Model Logical class diagram of data elements used in TLA data structures. 

DIv-3 Physical Data Schema Metamodels for 
e Enterprise Learner Profiles 
e Commercial 

o  XAPI/cmi5 IEEE P9274 
©  OpenBadge3.0/CTDL 
o RCD 
o —_LTI/OpenAPI/REST 
o LRMI modified to LRMI++ 
o OAuth/OICD 
OV-1 High Level Operational | The concept of TLA compliant enclaves, federations, and ecosystem and 
Concept engagement with stakeholders. 
OV-2 Operational Resource Ecology of operational nodes and information need lines (per DIV-1) in the 
Flow total learning ecosystem organized by engagement. 
OV-4 Organizational Stakeholders in the Total Learning Ecosystem organized by reporting 
Relationships structure. 
OV-6a Operational Rules Business rules for TLA governance structures. 
Model: Business Rules 
OV-6b Operational Rules Sequence of activities for roles and operational nodes. 
Model: State Transition 
SV-1 Systems Context Connections between technical elements within enclaves as nodes and 
Diagram federations between nodes. 
SvcV-1 Services Context Context of ecologies, enclaves, and federations within the TLA as data 
Diagram service contracts. 
SvcV-4 | Services Functionality High-level functions of each service and system component of the TLA. 
SvcV-5 Operational Activity to | Mapping of services to supporting use cases used to help migration 
Services strategies. 
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View | Description Purpose 


SvcV-10a | Services Business Rules | Business rules for TLA technical compliance. 
SvcV-10b | Services State Sequence of learner states that satisfy the continuum of andragogical, 
Transition Diagram pedagogical, and heutagogical/self-regulated learning use cases for the TLA. 
SvcV-10c | Services Event Trace Objective systems concept of execution showing five control loops for 
Diagram human performance. 
StdV-1 Standards Profile 2018/19 specifications and standards. 
StdV-2 | Standards Forecast Objective system policy, specifications, and standards alignment, including 
the proposed TLA standard. 
PV-1 Project Portfolio Timelines current horizon projects and efforts. 
Relationships 
PV-3 Project to Capability High-level alignment of projects to specification, and standard versions to 
enable Initial, Interim, and Final Operational Capability. 


1.1 AV-1 - Overview and Summary 
1.1.1 Architectural Description Identification 


This is an Architectural Description Document describing the TLA, an ecology of policy, specifications, 
and standards being developed by the ADL Initiative within the Force Education and Training (FE&T) 
component of the Office of the Undersecretary of Defense for Personnel and Readiness — OUSD(P&R). 
These documents are being developed for eventual inclusion in DoD Instruction (DODI) 1322.267. 


This ecology describes the constraints and enabling protocols, processes, and data contracts to create 
the future learning ecosystem, which is comprised of interoperable learning technologies that enable a 
concept for individual lifelong learning that enhances the human capital supply chain management 
process for human resources. 


The TLA represents a culmination of the ADL Initiative’s mission of modernization, policy development, 
and community outreach. The TLA Initial Operating Capability (IOC) is expected in Fiscal Year (FY) 2021, 
starting with mandatory inclusion of the Experience Application Program Interface (xAPI) specification in 
all new and legacy-upgrade training media, with a Final Operational Capability (FOC) in 2025. FOC will 
include a transition to Competency Based Learning (CBL), multi-level security (MLS), and digital badging. 
This initial architectural description is expected to be a living document that will be continually updated 
based on feedback and input from the ADL Initiative’s DoD stakeholders. 


1.1.2 Scope 


Among the “views” listed in Table 1, some are applicable to the objective state and some to the current 
2019 research effort as described. All are applicable to education- and training-focused elements of the 
DoD and the uniformed services. The views also identify DoD stakeholders outside of education and 
training organizations that may be interested in the TLA-generated data, or that may generate data that 
could be used to improve or enhance education and training, such as Manpower and Personnel (M&P), 
system acquisition, and unit operational readiness reporting agencies. The TLA does not prescribe how 
these external organizations structure, store or communicate their data. 


1 https://www.esd.whs.mil/Portals/54/Documents/DD/issuances/dodi/132226_dodi_2017.pdf?ver=2017-10-05- 
073235-400 
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1.1.3. Purpose and Perspective 


The ability to create, retain, and manage capable manpower is a critical national capability. It is in the 
interest of the DoD to maximize the available pool of trained soldiers, sailors, marines, and airmen, and 
to optimize their training and professional development throughout their careers. In addition to 
benefiting DoD stakeholders, this maximizes options for these personnel when they transition to non- 
DoD careers, serving to continually market DoD service as a rewarding and valuable career path. Thus, 
the TLA vision serves three complimentary functions: 


e Support for Lifelong Learning — provides rich, rewarding career experiences for service 
members to improve retention, accession, and self-actualization throughout their entire lives. 


e Human Capital Supply Chain Management — maintains a force structure and level of readiness 
that supports the National Defense Strategy as it evolves over time. 


e Alignment of DoD Learning Experiences — harmonizes DoD experience with the skills and 
competencies required in the civilian marketplace, preparing service members for transition to 
the civilian workforce. 


These high-level missions for the DoD’s education and training enterprise support the goals of the 
National Defense Strategy to improve lethality (by ensuring optimally qualified people in deploying 
units) and improve management functions (by reducing inefficiencies in the education and training 
pipeline). TLA-enabled enterprise analytics will improve the efficacy of training throughout the 
workforce pipeline and increase lethality with data-driven approaches to readiness. TLA compliant 
systems using competency-based learning will lower costs for training and reduce the size of the total 
force required to maintain the same operational posture. 


The TLA combines new learning tools, systems, activities, and content to establish a loosely coupled 
ecosystem made interoperable through common specifications, standards, and policy. The TLA includes 
models within this architecture that decompose to lower level capabilities and functions, and ultimately 
technical services, data, and interfaces. These enhance, optimize, or improve upon existing learning 
solutions, especially in reducing the time and labor overhead to provide education and training 
capabilities. In the future, it is envisioned that all learning experiences will interoperate within the future 
learning ecosystem. This includes traditional classroom activities, field trips, and digital learning, as well 
as simulators and on-the-job training experiences that make up a sizable portion of human learning. 


The TLA data strategy results in enhanced enterprise analytics that validate the efficacy of the education 
and training personnel receive. This data strategy was developed to accommodate future interfaces 
with other DoD systems (e.g., Manpower & Personnel, HR, Readiness) and key performance indicators in 
the operational environment. The TLA ecosystem will streamline the delivery and management of 
learning for four million uniformed and civilian members of the DoD, optimizing the supply of capable 
manpower to perform missions. 


This will be accomplished through DODI 1322.26 and its “fungible references” that define TLA specific 
and related commercial standards to aid in the adoption and migration of TLA functionality. Central to 
the TLA are the xAPI standard and the cmi5 specification. xAPI is an evolving IEEE standard designed to 
replace the legacy Shareable Content Object Reference Model (SCORM). The cmi5 specification builds 
upon the xAPI to provide an alternative way of packaging, delivering, and managing web-based 
instructional content using Learning Management Systems (LMS). 
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The xAPI and cmi5 specifications expose important learner data using a Learning Record Store (LRS), the 
server component of the xAPI. These learner data can be combined with other learner data to build a 
more comprehensive snapshot of learner proficiencies and what they should be focusing on next in 
support of both their career trajectory and DoD mission needs. The new TLA policy ecology will include: 


e Migrating legacy education and training solutions to leverage ubiquitous learning opportunities 
afforded by mobile technology, cloud computing, data analytics and other sensors. 


e Assessment and reporting of personnel knowledge, skill, ability, and other (KSAO) capabilities as 
competencies and credentials reportable to DoD manpower and personnel systems. 


e Developing standardized labeling for Government owned performance data for new system 
acquisitions related to human performance, education, and training. 


e Developing standardized labeling and curating of new and existing content used to provide future 
learning opportunities. 


e Forecasting of the personnel components of “system readiness” for fielded DoD units based on the 
status of their education and training. 


e Leveraging cloud-based computing and centralized IT as a service for network and server 
virtualization, as well as cybersecurity in multi-level security environments, to mitigate education 
and training technology obsolescence. 


e Providing operational security of the “capable manpower” critical national capability, implied by the 
potentially aggregated nature of learning data in context. 


e Providing governance of data, technical, and organizational policy and standards for critical 
education and training systems operating on the Global Information Grid. 


e Transitioning to a curation model from a content development model to enable significant reuse in 
the development of training events and opportunities. 


e Enhancing education and training product review cycles by integrating data related to learning 
content, activities, and competencies with readiness data. 


e Improving the throughput of learning resources by focusing only on gaps in a personal competency 
model instead of a standardized curriculum in the classroom model. 


e Providing feedback to developmental systems on human performance issues and populating finer 
grained personnel demand signals during the acquisition process. 


1.1.4 Context 


This Architectural Description Document serves to reduce the complexity of the TLA system of systems 
approach, decomposing the purpose, composition, operation, and maintenance of TLA components 
through the TLA policy framework. The figures below depict the logical, physical, functional, and 
operational characteristics of the objective end-state. They also identify near-term research objectives, 
associated test beds, and their impact on defining policy and informing technology investments. 


Each document is prepared from the viewpoint of stakeholders within the DoD education and training 
enterprise. While these views are prepared for specific stakeholders, they include potential linkages to 
other DoD functional areas such as M&P, system acquisition, logistics, and readiness reporting. 
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The TLA enables a learning ecosystem by defining commercial standards, technical specifications, and 
business rules that must be supported by learning tools, technologies, activities and content to conform 
with the TLA data strategy. These views should be considered as part of any education and training 
modernization effort. By exposing education and training data and providing the functions of learning as 
a series of microservices, the DoD can leverage the cost-effective models provided by cloud-based 
computing as the foundation for greater integration of systems over time to enable new capabilities. 


Federated Data Enables Enterprise Services 
* Common Course Catalog 
* Unique User ID 
* Enterprise Learning Analytics 


Federation 


| Federated Data 
Authority To Connect 


/ Numerous Learning sites, Mobile Learning, Training Aids, | 
/ Devices, Simulators, and Simulations (TADSS), and other / 
Ubiquitous Learning Activities create a / 


/ Federation 


/ 


a Ecology 
Figure 1. Future Learning Ecosystem. The entirety of enclaves and federations, and interfaces to ancillary systems 
and data, create the ecosystem. An enclave is a network boundary that encapsulates core data and systems for an 
organization related to education and training. Enclaves may federate their data with other enclaves through 
Authorities to Connect. Data ownership and authoritative systems are controlled by the parent organization. 


The FLE’s “ecology” includes the DODI 1322.26 policy document and its fungible references of 
specifications and standards and related policies (e.g. cybersecurity) that create the overall framework, 
as shown in Figure 1. The ecology is defined by three high-level concepts: 


The Enclave — is a set of computation and data assets for a DoD organization used to deliver and 
manage education and training opportunities. The enclave contains the core data, services, and 
technologies such as installed LMS solutions, learning devices, simulators, and back-end systems 
developed according to the TLA specifications and standards. An enclave will require a cybersecurity 
Authority to Operate (ATO) and will include all assets procured, installed, and managed at a 
particular “learning organization” within the DoD. 


The Federation — combines data from multiple enclaves, as well as truly mobile or ubiquitous 
learning devices that may follow a person through their career and connect as federates through a 
defined federation negotiation process. Federation creation requires a Memorandum of Agreement 
(MOA) and Authority to Connect (ATC) between enclaves. Federation negotiation allows for 
horizontal scaling of data lakes, and requires governance for managing data labeling, identity 
management, and metadata schemas and attribution. Cybersecurity requirements for “Zero Trust 


Networks” enable “bring your own device.” 
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e The Ecosystem — encompasses all enclaves or potential federations, as well as interfaces from 
systems outside of education and training that consume or produce data useful for learning (such as 
credentials reported to manpower and personnel systems). These external systems require 
additional MOAs and ATCs. The TLA policy framework includes the organizational processes for 
governance that manages interoperability and semantic consistency between TLA compliant 
elements within education and training. These external connections allow for complete human 
capital supply chain management, using federated users to maintain their own data standards and 
interface specifications. 


1.1.5 Status 


The draft TLA architecture baseline 0.5 version was developed in 2019. The 2018 Reference 
Implementation SV-1, SV-10, and Svc-4 views were previously published in the 2018 TLA report’. The CV- 
3, DIV-2, OV-1, and Svc-V1 were developed for the OUSD(I) Talent Development Toolkit (TDT) delivered 
in 2019. These draft architectural views included in the TLA DoDAF are being presented to solicit 
feedback and facilitate discussion for improvement before being submitted for consideration to the 
Defense ADL Advisory Committee (DADLAC) for inclusion among the DoDI 1322.26 fungible references. 


1.1.6 Tools and File Formats Used 


TLA DoDAF source documents are developed in Microsoft Word, Excel, PowerPoint, and Visio. The 
technical models are developed according to the semantics and notation of the Unified Modeling 
Language or System Modeling Language (UML/SYSML) using the Visio template. Schedules are 
developed in Microsoft Project or summarized in PowerPoint. Cross reference tables and matrices are 
developed in Microsoft Excel. The entire architecture is maintained on the ADL Initiative’s internal 
SharePoint site and will be transitioned to the ADL GitHub? as appropriate. 


1.1.7. Assumptions and Constraints 


The TLA assumes a requirement for data federations because of the horizontal scale of data* and 
because security enclave and access constraints across DoD components preclude a single “walled 
garden” solution. The TLA approach must also adhere to cybersecurity requirements and comply with 
the Defense Information Security Administration (DISA) policy and processes. 


It must provide normative guidance for program offices to establish, contract, and maintain its 
requirements through governance and provide a recapitalization strategy that enables the departments 
to plan, budget and transition without a negative impact to ongoing operations and mission. Initial 
fielding will be through pilot programs that field site-specific capabilities. Programs will expand and 
interconnect organically over time. 


1.1.8 Schedule 
FY 2019 


e September 2018 — Kickoff for OUSDI Talent Development Toolkit (TDT) requirements and 
architecture project 


? https://apps.dtic.mil/dtic/tr/fulltext/u2/1077398.pdf 
3 https://github.com/adinet/ 
4 ~4 million uniform and civilian personnel spending 2000+ hours a year training or in performance of their duties. 
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December 2018 — Interviews of IC stakeholders 

February 2019 — Commencement of the Navigator for Interoperable Learning Experiences (NILE), 
DATASIM, Data Analytics and Visualization Environment (DAVE), and the Privacy and Security for TLA 
(PS4TLA) projects 

March 2019 — Submission of OUSDI TDT requirements and Architecture Report 

August 2019 — Commencement of CASS authoring tools and CaSS requirements hardening projects, 
and Competency Framework Development (CFD) process documentation 

August 2019 — Commencement of the xAPI Profile Server and xAPI profiles for DoD use projects, and 
acceptance of the TLA portal software submission 

July-September 2019 — Integration of the 2019 TLA Reference Implementation 


FY 2020 


January 2020 — Commencement of the Competency and Skills System (CaSS) integration effort with 
the US Air Force Learning Services Ecosystem (AFLSE) project 

DRAFT DoDAF released to community for review and to solicit feedback 

xAPI as an approved IEEE standard 

RCD as an approved IEEE standard 

Commencement of LOM 2.0 Standards working group 

Finalize TLA specifications (interface specification, metamodels, business, rules, federation 
negotiation) 

NILE established as a TLA enclave — facilitate federated data testing and evaluation 

Advance metadata and data labeling 

Integrate Privacy & Security of Learner Information into Federated Identity, Credential, and Access 
Management (FICAM) approach defined by DISA/DoDCIO 

Establish Enterprise Course Catalog (sometimes referred to as Universal Course Catalog) 

Establish Universal Unique Identifier 

Establish accredited Learning Record Store (LRS) 

Establish cmi5 conformance testing and viewer capability 

CaSS IATT / IATO for Air Force, Army, and Navy 


FY 2021 


Initial Operational Capability (IOC) 

xAPI profile server operational 

cmi5 conformance test operational 

Federated activity and resource management services 
Enterprise Course Catalog operational 


FY 2025 


1.2 


Final Operational Capability (FOC) 


AV-2 Integrated Dictionary 


See Table 2. 
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Table 2. AV-2 Integrated Dictionary. 


Lexicon Term 
Accreditation 


Definition 

1. (M&S -V.V &A) The official certification that a model, simulation, or federation of 
models and simulations and its associated data is acceptable for use for a specific 
purpose. 

2. (RMF - ATO) The signed approval of a cybersecurity package under the Risk Management 
Framework (RMF) within DoD. 


Citation/Reference/Attribution 
https://csrc.nist.gov/projects/risk- 
management/risk-management- 
framework-(RMF)-Overview 


Activity and Resource 
Registry 


Manages the relationship between activities, competencies, Enabling and Terminal Learning 
Objectives, and other relevant information about each activity. Contains access information 
and permission to allow access to activities. 


https://apps.dtic.mil/dtic/tr/fulltext/u2/10 
77398.pdf 


person. 


Activity Statement A single xAPI record entry. https://in.nau.edu/its-lpd/xapi/ 
Activity Streams Time-stamped data ledger about learner experiences tracked across learning activities https://apps.dtic. mil/dtic/tr/fulltext/u2/10 
encountered by an individual or team. The Experience API (xAPI) communicates information |77398.pdf 
about each learning experience through a series of xAPI statements that encapsulate a 
learner's experience with different learning activities. 
Alignment 1. The positioning of the human capital system's policies, practices, and strategies in https://www.opm.gov/policy-data- 
relationship to the agency's strategic plan and performance plan, so what is done in the | Oversight/human-capital- 
system is in direct support of the agency's mission, goals, and objectives. management/reference-materials/ 
2. In Competency-based learning, it refers to the mapping of learning activities and other 
evidence of learner proficiency to a specific competency. A competency is encapsulated 
in a framework where each competency has a unique identifier, as well as its own 
variables and conditions that need to be satisfied to achieve different mastery levels. 
Aptitude 1. Anatural ability. https://www.merriam- 
2. Acomponent of a competency that relates to performing certain kind of work at a webster.com/dictionary/aptitude 
certain level. https://en.wikipedia.org/wiki/Aptitude 
Assertion 1. Astatement of fact or belief. https://www.edglossary.org/competency- 
2. In competency-based learning, it is a claim about a learner's mastery over a specific based-learning/ 
competency, or the elements that a competency is comprised of (e.g., behaviors, 
knowledge, skills, abilities, aptitudes, or other information). 
Assessment Wide variety of methods or tools that are used to evaluate, measure, and document the https://www.edglossary.org/assessment/ 
academic readiness, learning progress, skill acquisition, demonstrated proficiency, or 
educational needs of an individual or team. 
Attitude A psychological construct, a mental and emotional entity that inheres in, or characterizes a__| https://en.wikipedia.org/wiki/Attitude_(ps 


ychology) 


Authoritative Data 
Source 


A recognized or official data-production source which publishes reliable and accurate data 
for subsequent use by customers. The authoritative data may be the functional combination 
of multiple, separate data sources. 


https://definedterm.com/authoritative_da 
ta_source 
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Lexicon Term 
Bloom's Taxonomy 


Definition 
A set of three hierarchical models used to classify educational learning objectives into levels 
of complexity and specificity. 


Citation/Reference/Attribution 
https://en.wikipedia.org/wiki/Bloom%27s 
_taxonomy 


Communities of 
Practice 


A group of people bound by a shared interest, purpose, or practice where people share ideas 
and knowledge in many different ways, including real-time collaborative sessions, 
newsletters, chat forums, social groups, and links to websites. 


https://www.opm.gov/policy-data- 
oversight/human-capital- 
management/reference-materials/ 


Competencies 


The set of demonstrable characteristics and skills that are required by an individual or team 
for performance of a job. Characteristics include knowledge, skills, abilities, and other (KSAO) 
- such as attitudes, aptitudes, motivations and social elements. 


https://en.wikipedia.org/wiki/Competence 
_(human_resources) 


Competency Based 
Learning 


An instructional strategy that allows students to progress as they demonstrate mastery of 
academic content, regardless of time, place, or pace of learning. Competency-based 
strategies provide flexibility in the way that credit can be earned or awarded and provide 
students with personalized learning opportunities. 


https://www.ed.gov/oii- 
news/competency-based-learning-or- 
personalized-learning 


Competency 
Framework 


A model that broadly defines the blueprint for ‘excellent’ performance within an 
organization or sector. Generally, the framework will consist of numerous competencies, 
which can be applied to a broad number of roles within the organization or sector. 


https://participatoryartslearning.wordpres 
s.com/the-framework/what-is-a- 
competency-framework/ 


Reusable Competency 
Definitions 


A framework that describes the full range of competencies required to be successful in a 
specific occupation. It provides a formal representation of the key characteristics of a 
competency, independently of its use in any specific context. It enables interoperability 
among learning systems that deal with competency information by providing a means for 
them to refer to common definitions with common meanings. 


https://ieeexplore.ieee.org/document/444 
5693 


Competency 
Management System 


The system or platform that aligns evidence from education, training, or operational learning 
experiences to Competency Frameworks and asserts a predicted level of proficiency for 
individuals and teams. 


https://en.wikipedia.org/wiki/Competency 
_management_system 


Competency Model 


A data model for describing, referencing, and sharing competency definitions, primarily in 
the context of learning and development. Competency models are used to support key 
human capital programs such as selection, career development, training, and performance 
management. These models usually describe the required occupation-specific, or technical, 
competencies and general cross-occupational competencies (e.g., analytical competencies). 


https://www.opm.gov/policy-data- 
oversight/human-capital- 
management/reference-materials/ 


Competency Object 


An encapsulated set of related knowledge, skills, and abilities required to successfully 
perform "critical work functions" in a defined work setting. Competencies often serve as the 
basis for skill standards that specify the level of knowledge, skills, and abilities required for 
success in the workplace. 


https://sph.uth.edu/content/uploads/201 
2/01/Competencies-and-Learning- 
Objectives.pdf 


Computer Assisted 
Instruction (CAI) 


Learning presented, managed, and controlled by an instructor that is in a different classroom 
than the learner. The instructor is assisted by technologies that support communication and 
passing documents between the instructor and the learner. The instructor still scores the 
course and determines if the learner has mastered the TLOs. 


https://adminpubs.tradoc.army.mil/pamp 
hlets/TP350-70-3.pdf 
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Lexicon Term 


Computer Managed 
Instruction (CMI) 


Definition 

Learning presented, managed, and controlled by software (e.g., LMS). There is no travel. 
There is no instructor. There is no classroom. The computer software manages all course 
activities and their interactions with the learner. The computer software measures mastery 
by automated evaluation and reports learner scores to the student registration system or 
other learning system. 


Citation/Reference/Attribution 


https://adminpubs.tradoc.army.mil/pamp 
hlets/TP350-70-3.pdf 


Content Object 


An instance of a content class. While the class only defines the data structure, it is the 
content objects themselves that contain actual data. Once a content class is defined, several 
content objects / instances of that class can be created. 


https://doc.ez.no/eZ-Publish/Technical- 
manual/3.6/Concepts-and-basics/Content- 
management/The-content-object 


Content and Resource 


A set of processes and technologies that supports the collection, managing, and publishing 


https://en.wikipedia.org/wiki/Content_ma 


Management of information in any form or medium. When stored and accessed via computers, this nagement 
information may be more specifically referred to as digital content, or simply as content. In 
the TLA, content management registers, selects, reports and validates resources required for 
all learning opportunities. 
Context Refers to those objects or entities which surround a focal event, in these disciplines typically | https://en.wikipedia.org/wiki/Context_(lan 


a communicative event, of some kind. 


guage_use) 


Core Service 


Logic “wrappers” that provide services by managing requests in and out of data stores that 
can also federate to additional segments on multiple edge system. 


https://www.adlnet.gov/assets/uploa 
ds/TDT%20Report.pdf 


Course 


Aseries of learning activities about a specific subject that are sequenced together into a 
program of instruction (POI) or (Navy- Course of Instruction - COI). Each activity may include 
different types of instructional content. 


https://adminpubs.tradoc.army.mil/regula 
tions/TR350-70. pdf 


Course Management 
System (CMS) 


A collection of software tools providing an online environment for course interactions. A 
CMS typically includes a variety of online tools and environments. 


https://cft.vanderbilt.edu/guides-sub- 
pages/course-management-systems/ 


Credential 


An attestation of qualification, competence, or authority issued to an individual by a third 
party with a relevant or de facto authority or assumed competence to do so. 


https://en.wikipedia.org/wiki/Credential 


Credential Portability 


Portability means the credential has value locally, nationally or internationally in labor 
markets, education systems, or other trust-based systems. The learner or consumer uses the 
credential for a variety of purposes but the context and competencies the credential 
represents remain intact. A portable credential enables earners to move vertically and 
horizontally within and across the credentialing ecosystem for attainment of other 
credentials. 


https://connectingcredentials.org/wp- 
content/uploads/2016/06/Glossary-of- 
Credentialing-Terms.pdf 


Credentialing 


Formally grouped competencies and experiences used to define a capability or level of 
knowledge recognized across systems. Credentials in the TLA also refer to portable, digital 
badges that preserve the provenance of the conferrer and possess an evidentiary audit trail 
of assertions. 


https://www.accredible.com/credentials/ 
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Lexicon Term 
Curriculum 


Definition 

A set of courses constituting an area of specialization. All training conducted within a school, 
outlined into specific topics with detailed training objectives. Curricula are maintained as 
"programs of Instruction" (POI) or Courses of Instruction (COl). 


Citation/Reference/Attribution 
MIL-HDBK-29612 


Data Contract 


A formal agreement between a service and a client that abstractly describes the data to be 
exchanged. That is, to communicate, the client and the service do not have to share the 
same data types, only the same data contracts. A data contract precisely defines, for each 
parameter or return type, what data is serialized (turned into XML) to be exchanged. 


https://docs.microsoft.com/en- 
us/dotnet/framework/wcf/feature- 
details/using-data-contracts 


Data Dashboards and 
Analytics 


An information management tool that visually tracks, analyzes and displays key performance 
indicators (KPI), metrics and key data points to monitor the health of a business, department 
or specific process. 


https://www.klipfolio.com/resources/artic 
les/what-is-data-dashboard 


Data Model for Content 
Object Communication 


(IEEE Std 1484.11.1- 
2004) 


This standard describes the Computer Managed Instruction (CMI) data model that the DoD 
uses to support the interchange of agreed upon data elements and their values between a 
learning-related content object and a runtime service (RTS) used to support learning 
management. 


https://standards.ieee.org/standard/1484 
_11_1-2004.html 


Data Validation 


The process of ensuring data have undergone data cleansing to ensure they have data 
quality that is correct and useful. 


https://en.wikipedia.org/wiki/Data_validat 
ion 


Data Visualization 


Translating data from aggregated experiences and/or systems in order to provide a pictorial 
representation of the data analysis information collected to drive informed decisions. 


https://www.academia.edu/40046943/_2 
_TDT_Report 


Data-driven 


1. Decision making philosophy determined by or dependent on the collection or analysis of 
data. 

2. An organizational culture where data and information are the basis of all actions and 
gathering and analyzing of data is the core motivator. 


https://www.techopedia.com/definition/1 
8687/data-driven 


Decision Support 
System (DSS) 


An information system that supports decision-making activities. DSSs serve the 
management, operations and planning levels of an organization and help people make 
decisions about problems that may be rapidly changing and not easily specified in advance. 


https://en.wikipedia.org/wiki/Decision_su 
pport_system 


Digital Badge A validated token of accomplishment, skill, quality, or interest that can be earned in many https://openbadges.org/ 
learning environments. Digital badging allows a learner to display badges across the web. 

Distributed Learning By its nature, learning is distributed. With ubiquitous availability of mobile devices and other |DODI 1322.26 

(DL) technology advancements, Distributed Learning now includes any type of learning mediated 
with technology and accessed through a network or experienced via portable media. 

DoDI 1322.26 Establishes policy, assigns responsibilities, prescribes procedures, and establishes DODI 1322.26 


information requirements for developing, managing, providing, and evaluating Distributed 
Learning for DoD military and civilian personnel. Directs DoD to implement emerging 
interoperability specifications (e.g., xAPI) and formally charters DADLAC as oversight body. 


Ecosystem/Ecology 


A distributed, adaptive, and open system with properties of self-organization, scalability and 
sustainability inspired from natural ecosystems. 


https://en.wikipedia.org/wiki/Digital_ecos 
ystem 
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Lexicon Term Definition Citation/Reference/Attribution 

Edge System Modern network architecture defines edge systems as independent systems that sit on the __| https://itknowledgeexchange.techtarget.c 
outside periphery or edge of the network. The separation between TLA core and edge om/modern-network-architecture/what- 
systems provides layered security boundaries around core business systems. is-an-edge-system/ 

Education Developing general knowledge, capabilities, and character through exposure to and learning |https://www.esd.whs.mil/Portals/54/Docu 
of theories, concepts, and information. Education is traditionally delivered by an accredited | ments/DD/issuances/ai/a040p.pdf?ver=20 
institution and may relate to a current or future mission-related assignment. 17-07-19-142854-093 

Efficiency The ratio of the outcome or output to the input of any program; the degree to which https://www.opm.gov/policy-data- 
programs are executed or activities are implemented to achieve results while avoiding oversight/human-capital- 
wasted resources, effort, time, and/or money. management/reference-materials/ 

e-learning learning utilizing electronic technologies to access educational curriculum outside of a http://www.elearningnc.gov/about_elearn 


traditional classroom. In most cases, it refers to a course, program or degree delivered 
completely online. 


ing/what_is_elearning/ 


Enabling Learning 
Objective (ELO) 


Precise three-part statement ... describing what the learner can accomplish in terms of the 
expected student performance under specific conditions to accepted standards. 


https://adminpubs.tradoc.army.mil/pamp 
hlets/TP350-70-3.pdf 


Enclave Within the context of the TLA, an enclave is a segregated sub-network managed by a DoD https://apps.dtic.mil/dtic/tr/fulltext/u2/10 
component (e.g., Defense Acquisition University, Air Education and Training Command), 77398.pdf 
located behind a firewall with managed access to support education and training activities. 

Evidence Anything presented in support of an assertion of competency. https://en.wikipedia.org/wiki/Evidence 


Experience Application 
Programming Interface 
(xAPI) 


A software specification that provides a standardized format for communicating and tracking 
all types of learning experiences across the enterprise. Learning experiences are recorded in 
a Learning Record Store (LRS), the server-side implementation of the xAPI. The xAPI uses 
Statements as a communication mechanism. Statements are evidence of learner 
performance that include [Actor --> Verb --> Object]. 


https://adInet.gov/research/performance- 
tracking-analysis/experience-api/ 


Expertise Levels 


A theoretical explanation for understanding how adults acquire skill and transition from 
being a novice to an expert (e.g., the five stages of skill acquisition: novice, advanced 
beginner, competent, proficient, and expert). 


https://www.bumc.bu.edu/facdev- 
medicine/files/2012/03/Dreyfus-skill- 
level.pdf 


Failure Modes and 
Effects Analysis (FMEA) 


One of the first highly structured, systematic techniques for failure analysis. It was 
developed by reliability engineers in the late 1950s to study problems that might arise from 
malfunctions of military systems. An FMEA is often the first step of a system reliability study. 
It involves reviewing as many components, assemblies, and subsystems as possible to 
identify failure modes, and their causes and effects, including human performance. 


https://en.wikipedia.org/wiki/Failure_mod 
e_and_effects_analysis 


Federation 


A group of computing or network providers agreeing upon standards of operation ina 
collective fashion. 


https://en.wikipedia.org/wiki/Federation_( 
information_technology) 


Future Learning 


A transformation away from disconnected, episodic experiences and towards a curated 


https://www.adInet.gov/modernizing- 


Ecosystem continuum of lifelong learning, tailored to individuals, and delivered across diverse locations, |!earning 
media, and periods of time. 
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Lexicon Term 
Governance 


Definition 

Establishment of policies with continuous monitoring of their proper implementation, by 
members of a governing body. Governance includes the mechanisms required to monitor 
and enforce compliance. 


Citation/Reference/Attribution 


http://www. businessdictionary.com/defini 
tion/governance.html 


Higher Order Cognition 
/ Higher Order Thinking 


1. Acomposition of sophisticated neurodevelopmental functions that influence decision 
making including concept acquisition, systematic decision making, evaluative thinking, 
brainstorming, and rule usage. 


2. Higher-order thinking is a concept of education reform based on the concept that some 


types of learning require more cognitive processing than others. These include concepts 
that require strategic thinking, planning, or comprehending the interconnectivity of 
concepts, places, events, and people. 


https://www.sciencedirect.com/topics/psy 
chology/higher-order-cognition 


Human Capital Supply 
Chain 


Refers to the integration of business planning, strategic workforce planning, staffing and 
recruiting processes and technology to enhance organizational productivity. 


https://en.wikipedia.org/wiki/Human_Capi 
tal_Supply_Chain 


Identity Management 


Also known as identity and access management (IAM). IAM refers to a framework of policies 
and technologies for ensuring that the proper people in an enterprise have the appropriate 
access to technology resources. 


https://en.wikipedia.org/wiki/Identity_ma 
nagement 
OUSDI Final Reort-JAGFinal.docx 


Inference 


In the TLA and competency-based learning, an inference is a calculation of the strength of a 
relationship between two concepts based on statistical reduction of cause and effect data. 
An inference considers steps in reasoning, moving from premises to logical consequences. 


https://en.wikipedia.org/wiki/Inference 
OUSDI Final Reort-JAGFinal.docx 


Instructional Strategies 


Instructional Theories provide insights about what is likely to happen and why with respect 
to different kinds of teaching and learning activities while helping indicate approaches for 
their evaluation. Strategies based on a chosen instructional theory offer explicit guidance on 
how to better help people learn and develop. Instructional designers focus on how to best 
structure material and instructional behavior to facilitate learning. 


https://en.wikipedia.org/wiki/Instructional 
_theory 


JavaScript Object 
Notation (JSON) 


An open-standard file format that uses human-readable text to transmit data objects 
consisting of attribute—value pairs and array data types (or any other serializable value). 


https://en.wikipedia.org/wiki/JSON 


Job, Duty, Task Analysis 
(JDTA) 


A standard process for capturing pertinent data that describes work performed for a specific 
job. This includes the decomposition and structuring of work and the application of 
appropriate attributes to the work, at the task level, to more comprehensively describe the 
work. 


https://www.public.navy.mil/netc/ile/doc 
uments/NETCResource/NAVEDTRA_137.P 
DF 


Just-in-Time Teaching 
(JITT) 


Providing access, anytime and anywhere, to learning and training that is the relevant 
information in the correct moment. Just-in-time training is not something that takes 
separate time from day-to-day work, instead is something that happens throughout the 
working day easily and quickly. 


https://www.swiftelearningservices.com/i 
mprove-learning-culture-by-embracing- 
just-in-time-training-in-workplace/ 


Knowledge, Skills, and 
Abilities (KSAs) 


The attributes required to perform a job and are generally demonstrated through qualifying 
experience, education, or training. Knowledge is a body of information applied directly to 
the performance of a function. Skill is an observable competence to perform a learned 


https://www.opm.gov/policy-data- 
oversight/classification- 
qualifications/general-schedule- 
qualification-policies/#url=exp 
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Lexicon Term 


Definition 
psychomotor act. Ability is competence to perform an observable behavior or a behavior 
that results in an observable product. 


Citation/Reference/Attribution 


Learner 


The generic term for a user in a distributed learning environment that generates data while 
interacting with elements within that environment. 


https://github.com/adI|net/xAPI- 
Spec/blob/master/xAPI-About.md 


Learner Profile, Learner 
Profile Service 


An adaptive data structure that tracks and stores learning and skills mastery over a lifetime 
of learning experiences. The information can be used to make academic or employment 
decisions, among other things. 


https://www.armyupress.army.mil/Portals 
/7/journal-of-military- 
learning/Archives/jml-april-2017-whole- 
book.pdf 


Learning Activity 


Activities designed or deployed by the teacher to bring about or create the conditions for 
learning. 


http://newlearningonline.com/learning- 
by-design/glossary/learning-activity 


Learning Activity 
Provider 


An activity provider is any tool or system that generates data about learning experiences, 
achievements, and job performance and sends it to the LRS. 


https://xapi.com/ecosystem/ 


Learning Content 
Management System 
(LCMS) 


An environment where developers can create, store, reuse, manage and deliver learning 
content from a central object repository, usually a database. LCMS generally work with 
content that is based on a learning object model. 


http://edutechwiki.unige.ch/en/LCMS 


Learning Event 
Management 


The process of recording learning data the order that it is done and determining which 
learning activity a learner should experience next. In the TLA, this is the function that 
schedules, initiates, monitors and closes out each learning event. It provides the control logic 
around the LRS. Previously called “activity management.” 


https://en.wikipedia.org/wiki/Activity_ma 
nagement 
OUSDI Final Reort-JAGFinal.docx 


Learning Management 
System (LMS) 


A software application that controls the administration, documentation, tracking, reporting, 
and delivery of educational courses, training programs, or learning and development 
programs. 


https://en.wikipedia.org/wiki/Learning_m 
anagement_system 


Learning Record Store 
(LRS) 


A server-based system capable of receiving and processing web requests, that is responsible 
for receiving, storing, and providing access to Learning Records.” The LRS is designed to 
enable systems to store and retrieve xAPI statements, store xAPI state, and store various 
other xAPI metadata from other systems. 


https://github.com/adI|net/xAPI- 
Spec/blob/master/xAPI-About.md 


Learning Science 


An interdisciplinary field that works to further scientific, humanistic and critical theoretical 
understanding of learning as well as to engage in the design and implementation of learning 
innovations, and the improvement of instructional methodologies. 


https://en.wikipedia.org/wiki/Learning_sci 
ences 


Level of Mastery 


Refers to how good or proficient you have to be in an attribute, trait, or skill to use it in a job, 
or proceed to the next level. 


https://en.wikipedia.org/wiki/Mastery_lea 
rning 


Lifelong Learning 


An acknowledgement that learning is pervasive. It encompasses a “continuum of learning” 
from birth to adulthood that connects formal and informal learning experiences into a 
cohesive set of competencies and credentials to drive future educational and work choices. 


https://www.adInet.gov/modernizing- 
learning 


Linked Data 


A method of publishing structured data so that it can be interlinked and become more useful 
through semantic queries. 


https://en.wikipedia.org/wiki/Linked_data 
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Lexicon Term 
Live, Virtual, 
Constructive, Game 
(LVCG) 


Definition 
Integrates emerging simulations and technology-based exercises into unit and Soldier 
training and is part of the Army's Integrated Training Environment. 


Citation/Reference/Attribution 


https://www.army.mil/standto/archive_20 
13-05-23 


Master Scenario Events 


Provides a timeline and location for all expected exercise events and injects (actions that 


https://emilms.fema.gov/IS130a/groups/2 


List (MSEL) push the scenario forward). 5.html 
Measures of Outcome metrics that define the level of achievement for a set of goals and the intended http://acqnotes.com/acqnote/tasks/meas 
Effectiveness (MOE) results. They may be expressed as probabilities or key performance indicators that a job or ures-of-effectivenessrequirements 


task has been performed to standard. 


Measurement of 
Performance (MOP) 


A criterion used to assess friendly actions that are tied to measuring task accomplishment. 


https://www.jcs.mil/Doctrine/Joint- 
Doctrine-Pubs/3-0-Operations-Series/ 


Metadata 


Data that describe other data. Metadata describes learning resources, competencies, 
learners, and data that is used within the TLA. It is often used for discovery and 
identification of resources using a search tool. Similarly, a Competency Management System 
will use metadata to predict proficiency levels upon receiving an assertion from a specific 
learning activity. 


https://techterms.com/definition/metadat 
a 


Micro Credentials 


Limited certifications that represent an individual's demonstration of specific skills or 
knowledge. They are sometimes conferred as open or digital badges as opposed to a more 
traditional written diploma. 


http://policyatlas.org/wiki/Micro- 
credentials 


Microlearning 


A way of teaching and delivering content to learners in small, very specific bursts. The 
learners are in control of what and when they’re learning. 


https://elearningindustry.com/why- 
microlearning-is-huge 


Microservice 


A software development technique—a variant of the service-oriented architecture (SOA) 
architectural style that structures an application as a collection of loosely coupled services. 


https://en.wikipedia.org/wiki/Microservice 
s 


Mobile Learning 


Mobile Learning focuses on learning across contexts and locations by the means of mobile 
devices (e.g. laptops, smart phones, personal digital assistants, game devices, tablet PCs, and 
e-books). Mobile devices are used to access online courses and resources and can also foster 
collaboration among individuals, conduct assessments and evaluations, provide access to 
performance support, and capture evidence of a learning activity. 


https://www.opm.gov/WIKI/training/Lever 
aging-New-Technologies-for-Employee- 
Development-Programs.ashx 


Needs Assessment 


A systematic process for determining and addressing needs, or "gaps" between current 
conditions and desired conditions. A needs assessment is the process of identifying the "gap" 
between performance required and current performance. 


https://www.opm.gov/policy-data- 
oversight/training-and- 
development/planning-evaluating/ 


Observer, Instructor, 
Controller, Supervisor 
(OICS) 


A collective identification of roles in the Total Learning Architecture which provide 
mentoring, guidance or approval for learning activities and evaluations of learners. 


TLA DODAF 


On the Job Training 
(OJT) 


A form of training provided at the workplace. Employees also get a hands-on experience 
using machinery, equipment, tools, materials, and facing the challenges that occur during 
the performance of the job. 


https://en.wikipedia.org/wiki/On-the- 
job_training 
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Lexicon Term Definition Citation/Reference/Attribution 
Paradata Usage data about learning resources that include quantitative metrics (e.g., how many times |https://en.wikipedia.org/wiki/Paradata_(le 
a piece of content was accessed) and pedagogic context, as inferred through the actions of | arning_resource_analytics) 
educators and learners. Paradata may be operationalized as a specific type of metadata, 
however the construct differs from traditional descriptive metadata that classify the 
properties of the learning resource itself, and instead involves the capture—and open 
resharing—of in situ information about online users’ interactions related to the resource. 
Performance The process of collecting, analyzing and/or reporting information regarding the performance |https://en.wikipedia.org/wiki/Performanc 
Measurement of an individual, group, organization, system or component. e_measurement 
Persona A Persona is a component of Identity Management that expresses the context for an https://github.com/adInet/xAPI- 


individual’s social interactions with work, family, friends, etc. Each persona has a different 
set of data, associations, or other attributes. For example, an individual's Facebook account, 
work account, and gym membership are all separate personas of the same person. 


Spec/blob/master/xAPI-About.md 


Personalization 


A diverse variety of educational programs, learning experiences, instructional approaches, 
and academic-support strategies that are intended to address the distinct learning needs, 
interests, aspirations, or cultural backgrounds of individual students. 


https://www.edglossary.org/personalized- 
learning/ 


Personally Identifiable 
Information (PII) 


Any representation of information that permits the identity of an individual to whom the 
information applies to be reasonably inferred by either direct or indirect means. 


https://www.dol.gov/general/ppii 


Proficiency 1. Army: Proficiency is the completeness of achievement of lower level standards to an ADP-7 
overarching standard. NAVEDTRA 137 
2. Navy: Proficiency is equivalent to level of mastery. MIL-HDB-29612 
3. Joint: Proficiency is meeting standard. 
Profile A set of rules and human- and/or machine-readable documentation of application-specific _/ https://github.com/adInet/xapi-profiles 
vocabulary concepts, statement patterns, extensions, and statement templates used when 
implementing xAPI in a specific context. 
Readiness The ability of US military forces to fight and meet the demands of the National Defense https://www.militaryfactory.com/dictionar 


Strategy. Readiness is the synthesis of two distinct but interrelated levels. 

e Unit Readiness is the ability to provide capabilities required by the combatant 
commanders to execute their assigned missions. This is derived from the ability of each 
unit to deliver the outputs for which it was designed. 

e Joint Readiness is the combatant commander’s ability to integrate and synchronize 
ready combat and support forces to execute his or her assigned missions. 


y/military-terms- 
defined.asp?term_id=4430 


Recommender Service 


A subclass of information filtering system that seeks to predict which learning activity a 
learner should interact with to further career goals or to meet operational needs. 


https://en.wikipedia.org/wiki/Recommend 
er_system 


Representational State 


A software architectural style that defines a set of constraints to be used for creating Web 


https://en.wikipedia.org/wiki/Representati 


Transfer (REST) services. Web services that conform to the REST architectural style, termed RESTful Web onal_state_transfer 
services (RWS), provide interoperability between computer systems on the Internet. 
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Lexicon Term 


Resident Instruction 
(Live) 


Definition 

Education or training activities that are presented, managed, and controlled by an onsite 
instructor or facilitator. There is an instructor, a classroom, and one or more learners in the 
same classroom at the same time. 


Citation/Reference/Attribution 
https://sill- 
www.army.mil/DOTD/divisions/pdd/docs/ 
Army%20Learning%20Model%202015.pdf 


Return on Investment 
(ROI) 


A performance measure that compares the monetary value of the business impacts gained 
from a project or a program with the actual cost of implementing the project or program. 


https://www.opm.gov/policy-data- 
oversight/human-capital- 
management/reference-materials/ 


Shareable Content 
Object Reference 
Model (SCORM) 


A set of technical standards for eLearning software products developed in the early 2000s. 
Specifically, SCORM governs how online learning content and Learning Management 
Systems (LMSs) communicate with each other. 


https://adInet.gov/scorm 


Self-Directed Learning 


Learners identify potential for learning from novel experiences and determine own path 
(Heutagogy). 


https://www.teachthought.com/pedagogy 
/a-primer-in-heutagogy-and-self-directed- 
learning/ 


Self-Regulated Learning 


Refers to one’s ability to understand and control one’s learning environment. The self- 
regulatory processes that learners apply to transform their cognitive abilities into academic 
performance. Self-regulation includes goal setting, self-monitoring, self-instruction, and self- 
reinforcement. 


https://lincs.ed.gov/sites/default/files/3_T 
EAL_Self%20Reg%20Learning. pdf 


Skill Decay 


The loss of knowledge or a skill following a period of nonuse. 


https://www.researchgate.net/publication 
/232904872_Factors_That_Influence_Skill 
_Decay_and_Retention_A_Quantitative_R 
eview_and_Analysis 


Social Learning 


A theory of learning process and social behavior which proposes that new behaviors can be 
acquired by observing and imitating others. Using technology to learn vicariously through 
others’ experiences, benefit from their knowledge and ability to highlight important 
information, and to increase interest and enjoyment in learning. 


https://www.psychologytoday.com/us/bas 
ics/social-learning-theory 


Specifications and 
Standards 


A set of community-accepted or normative technical rules used to coordinate 
interoperability of systems or interchanges of data. 


https://www.dsp.dla.mil/Specs-Standards/ 


Stakeholder 


An individual, or group of individuals, who have a significant or vested interest in the 
outcomes of ongoing education and training modernization efforts. 


https://www.opm.gov/policy-data- 
oversight/human-capital- 
management/reference-materials/ 


Subject Matter Expert 


A person with bona fide expert knowledge about what it takes to do a job. First-level 
supervisors are normally good SMEs. 


Delegated Examining Operations 
Handbook 


Talent Development 


1. Building the knowledge, skills, and abilities of others and helping them develop and 
achieve their potential so that the organizations they work for can succeed and grow. 

2. Talent Development cultivates a continuous learning and development environment to 
ensure that an agencies workforce can adapt to globalization, internal restructuring, and 
adaptations that affect how work is performed. 


1. https://www.td.org/insights/talent- 
development 

2. https://www.opm.gov/policy-data- 
oversight/human-capital- 
management/talent-management/ 
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Lexicon Term 
Talent Development 
and Training 


Definition 

Talent Development and Training is the creation and delivery of learning resources and 
opportunities increasing the employees’ capacity to successfully perform in their roles and 
advance their careers. 


Citation/Reference/Attribution 
https://www.opm.gov/services-for- 
agencies/hr-line-of-business/enterprise- 
architecture/brm_report_v2.pdf 


Talent Development 
Planning 


Talent Development cultivates a continuous learning and development environment to 
ensure that an agency’s workforce can adapt to globalization, internal restructuring, and 
adaptations that affect how work is performed. To integrate Talent Development with Talent 
Management an analysis of workforce data is needed to determine how an agency will meet 
future needs through the development or cross-training of talent who possess the required 
skills. 


Definition provided by Shanaz Porter 
(OPM) 


Talent Management 


1. Anorganization's ability to recruit, retain, and produce the most talented employees 
available in the job market using data to inform purposeful recruitment and personnel 
selection, assignment, and management. 

2. Asystem that promotes a high-performing workforce, identifies and closes skills gaps, 
and implements and maintains programs to attract, acquire, develop, promote, and 
retain quality and diverse talent. 


https://www.opm.gov/WIKI/training/Talen 
t-Development-Glossary.ashx#T 


Talent Management 
System 


A system that addresses competency gaps, particularly in mission-critical occupations, by 
implementing and maintaining programs to attract, acquire, develop, promote, and retain 
quality talent. 


https://www.opm.gov/policy-data- 
oversight/human-capital- 
management/reference-materials/ 


Technical 
Interoperability 


A characteristic of a product or system, whose interfaces are completely understood, to 
work with other products or systems, at present or in the future, in either implementation or 
access, without any restrictions. 


https://en.wikipedia.org/wiki/Interoperabi 
lity 


Terminal Learning 
Objective (TLO) 


Specifies what learner should know or be able to do at the end of a course that they didn't 
know or couldn’t do before taking the course. Each TLO must have at least one ELO. 


https://adminpubs.tradoc.army.mil/pamp 
hlets/TP350-70-3.pdf 


Total Learning 
Architecture 


A research and development activity sponsored by the ADL Initiative and conducted in 
collaboration with stakeholders from across the defense community, professional standards 
organizations, and commercial industry. The TLA is a collection of specifications for accessing 
and making use of lifelong learning-related data to enable next-generation learning. 


https://www.adinet.gov/tla/ 


Training 


Process of placing or enrolling an employee in a planned, prepared, and coordinated 
program, course, curriculum, subject, system, or other education activity that improves 
individual and organizational performance and assists in achieving the agency’s mission and 
performance goals. 


https://www.esd.whs.mil/Portals/54/Docu 
ments/DD/issuances/ai/a040p.pdf?ver=20 
17-07-19-142854-093 


Universal Unique 
Identifier (UUID) 


A 128-bit number used to identify information in computer systems. 


https://en.wikipedia.org/wiki/Universally_ 
unique_identifier 


User Interface/User 
Experience (UI/UX) 


The space where interactions between humans and machines occur. 


https://en.wikipedia.org/wiki/User_interfa 
ce 
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2.0 CAPABILITY VIEWS (CV) 


Capability views describe the TLA from the perspective of measurable improvements to the education 
and training enterprise. It is stated in terms of value added to the national defense strategy. 


2.1 CV-1 Vision 


The TLA vision shows the alignment of the phased deployment of capabilities aligned with the lines of 
effort in the ADL Initiative research portfolio. These lines of effort and related capabilities, specifications 
and projects are listed in Table 3. 


2.2 CV-2 Capability Taxonomy 


Table 2 describes the hierarchy of capabilities provided, and measurements of effectiveness to 
determine the impact of the new TLA capability on lethality or management systems in the education 
and training enterprise. 


2.3 CV-3 Capability Phasing 


The TLA capabilities will be phased-in as a series of migration efforts. Existing systems and content can 
be recapitalized by exposing additional interfaces and conversion of data structures or migrating to new 
data repositories. Eventually the adoption of services or microservice-based systems can replace legacy 
systems as part of obsolescence management and provide even more capabilities as end-to-end 
management and optimization of the human capital supply chain is enabled. This is shown in Figure 2 as 
the TLA Capability Maturity Model. 


2.4 CV-6 Capabilities to Operational Activities Mapping 


Table 4 presents a cross reference between the capabilities listed in Table 2 and stakeholder processes 
(i.e. “operational activities”) that support or are supported by those capabilities. 
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Table 3. CV-1 TLA Vision and Alignment with ADL Initiative Lines of Effort. 


TLA Maturity Level | Alignment with NDS Capability Provided 
Decouple learning solutions from Vendor 
1 
Expansion of Instrumented Learning Activities 
Legacy Content Recapitalization 
2 
Enhanced Enterprise Analytics 
Reduced Schooltime Duty Cycle* 
Integration with Occupational Alignment 
3 
Operational Data Validation 
Full Integration with CDO Data Interoperability 
4 . ; . 
Incorporation of Learning Adaptation Support 
5 F ; 
Fully Automated Human Capital Demand Signals 


ADL Project Supporting 


PASTLA-> Privacy Api ->Accredited LRS 
Profile Server 


Pub/Sub-> xAPI MOM ->Activity Management Service-> DATASIM 


cmi5 viewer 


Content and Resource Registration Service 
Federated Activity Indices ->Data Labeling->Enterprise Course Catalog (ECC) 


Flash Deprecation metadata 


OUSDI->Federated Identity Management -> UUID 
CaSS AFLSE-> CaSS hardening ->NILE-> Data Labeling->CBL 


Associated Specification 


xAPI, cmi5, MOM IEEE P9274.3.1, 
REST 


Federation Negotiation (TLA Spec), 
NIST 800 (ZTN), IAM (DISA), LTI, 
Service Specs for TADSS and 


support data, LRMI 
LRMI,DODI 1322.26 


Federated Negotiation (TLA Spec), 


ZTN, IAM, xAPI, TLA Business Rules 
IEEE P1484.20.1 RCD 


Expected Time to Complete 


Airman Learner Record, ELRR 


LEGEND 
Threshold for TLA IOC FY 221 


ADL Policy or origin 


External Policy, Spec or standard 


Improve Management Systems 


loc 
Improve Management Systems 
and Increase Lethality 
Increase Lethality 
FOC 


Associated Specifications rolled 
out as fungible references in 
DODI(D) 1322.26 via DADLAC 


Enterprise Learner Record Repository (ELRR metamodel p 

MILGEARS Interoperability MILGEARS API Sep-23 

Digital Badging-> Centralized Credential management OPM Credential Policy Sep-23 

Data Governance-> Readiness Data Integration DRRS API, etc. Sep-25 
DEERS API, etc. (e.g. HR system 

Data Governance-> M&P/HR data integration update TBD) Sep-25 
GEIR, S-1000D etc., xAPI, 

Data Governance-> EPSS/IETM Integration Federation Negotiation Sep-25 
Model Based Product Support, ISO 

Data Governance-> ILS/Product Support Data Integration 10303 Sep-25 

Recommenders** TLA Interface Spec, Business Rules |Outyear TBD 

Internet of Learning Things ** TLA Interface Spec, Business Rules |Outyear TBD 

Meta Adaptation ** TLA Interface Spec, Business Rules |Outyear TBD 

Machine Learning ** TLA Interface Spec, Business Rules |Outyear TBD 


Project 
NATO CMM stakeholder engagement 
DADLAC stakeholder engagement 


Backend Infrastructure and Outreach | Virtualization Management/Hosting/Sandbox 


experimentation to vet concepts for inclusion in TLA policy framework 


Support Learning Warehouse 


XAWG, TLAWG, Ltech reform, xAPI std 


clearing house for product Transition 


Conformance Suite 
Curated GitHub 
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Outreach and marketing 
Maintain existing user base 


Maintain existing user base 


* Competency based learning 
enhances lethality by tying 
credential back to demonstrated 
work performance, and enables 
shortening of formal education 
cycles so more time is spent 
operational/deployable 
-> Follows 


Underlined are IRAD projects 


Italicized are candidates for 


transition to separate Program of 
Record (PoR) 


** Not currently in portfolio 


Table 4. CV-2 Capability Taxonomy. 


be =AV7-)| 


1.1 Level 


1.1.1 Level 


Capability Maturity 


ID 


Purpose and Description 


Measures of Effectiveness 


Optimize the 
selection, 
retention, 
and 
assignment 
of capable 
manpower 
through 
enhanced 
education 
and training 


Level 


Decouple learning Provide central point for records from 1.1.1 IOC (1) Allows disparate LMS, CMS and other instrumented activities to record xAPI in central LRS without Reduce overhead costs to maintain training 
systems from disparate CMS/LMS requiring interface between internal system components or data. 
proprietary vendor Create agility with legacy LMS license 1.1.2 Reduce financial burden of maintenance at sites by removing dependencies between content, LMS, Reduce overhead costs to maintain training 
technology management fees assessment and other institutional administrative functions. 
Expansion of Address full range of formal and informal 1.2.1 Include the assertion of competency that comes from demonstration in work context of knowledge, Demonstration of performance data capture 
Instrumented learning, including on the job experiences skills, abilities and other characteristics (KSAOs), especially those associated with motivation and soft for OJT, LVCG, traditional eLearning, and 
Learning Activities skills - combination of Personnel and Training data. attributes typically aligned with 
and integration with performance evaluations 
learning data Enable Self-Paced Learning in common 1.2.2 Manage adult learning activities using same data structures as formal pedagogical setting. Reduce overhead costs to maintain training 
system 
Enable Self Directed Learning in common 1.2.3 Manage heutogical learning activities using same data structures as formal pedagogical setting. Reduce overhead costs to maintain training 
system 
Legacy Content Eliminate redundancy in course 1.3.1 Identify courses and course elements that are redundant between agencies and remove them from Improve content recapitalization ROI 
Recapitalization development sustainment to improve cost performance of courseware maintenance activities. 
Evaluate frequency of use of activities 1.3.2 IOC (2) Enhanced curriculum review by determining which activities are used most often. Improve content recapitalization ROI 
Correlate user experience feedback 1.3.3 Enhanced curriculum review by determining which activities are most popular or yield best Improve content recapitalization ROI 
experiences. 
Evaluate content efficacy 1.3.4 Enhanced curriculum review by determining which activities are correlated to superior performance. Improve content recapitalization ROI 
Support recapitalization of legacy learning 1.3.5 Allow integration of existing LMS, simulation systems, HR databases, operational data, offline Improve content recapitalization ROI 
technology (LMS, etc.) assessment systems and network management systems without requiring extensive refactoring of 
solutions. 
Enhanced Identify systemic performance issues 1.4.1 3 Enhanced curriculum review by determining which areas of competency, curriculum, or content are Reduce cycle time for curricular review 
Local/Enterprise associated with system service wide performance problems. 
Learning Analytics Optimize individual path for completing 1.4.2 Improve student opportunity to pass class/excel at learning event. Reduce cycle time for curricular review 
course/learning event Reduce overhead costs to maintain training 
Optimize individual credential achievement | 1.4.3 Improve student time required to achieve degree/certification. Reduce cycle time for curricular review 
Reduced overhead costs to maintain training 
Optimize requirements of Current Job 1.4.4 Improve alignment of overall learner skillset and proficiency with job requirements. Improve overall unit readiness 
Improve student performance 1.4.5 Maximize pass rate for students enrolled in course (pump, not filter). Reduce overhead costs to maintain training 
Reduced Schooltime | Maximize training site throughput 1.5.1 Maximize number of learners qualifying per period (not linked to time-based schedule but based on Reduce overhead costs to maintain training 
Duty Cycle contention of learning resources to quickly achieve qualification). 
Improve speed of qualification and 1.5.2 Maximize number of qualified personnel who can perform a duty by reducing time to qualify (improve Improve overall unit readiness 
proficiency "operational duty cycle"). 
Optimize current career trajectory 1.5.3 Ensure knowledge and skill acquisition aligns with future job requirements. Improve overall unit readiness 
Integration of low- Support recapitalization of legacy 1.6.1 3 Allow integration or synchronization with ongoing improvement initiatives to leverage ongoing capital Improve content recapitalization ROI 
level competency improvement initiatives investment. 
data with Support career transition 1.6.2 3 Improve attractiveness of Military as a career option /improve retention. Improve overall retention 
Occupational Provide single authoritative reference point | 1.6.3 3 Be able to find competency and credential data on any personnel without having to manually search All DOD personnel are accessible with 
Alignment for credentials and competencies through multiple systems. records >95% complete 
Select new career 1.6.4 3 Determine possible options for career change based on legacy skills inventory, motivation and aptitude | Improve overall retention 
aligning with other jobs' requirements 
Operational Data Tie human performance objectives to 1.7.1 FOC (4) Reduce cycle time for curriculum review from 3 years to continuous improvement and relevancy. Reduce cycle time for curricular review 
Validation curricular requirements 
Provide reliable predictions of overall 1.7.2 Improve estimation of warfighting capability by increasingly granularity of human contribution to Improve overall unit readiness 
warfighting readiness based on readiness. 
competency state 
Maximize options and fitness for detailing 1.7.3 Ensure demand signals for future personnel requirements are transmitted and addressed as early as Improve overall unit readiness 


possible in the supply chain. 
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1.0 Level 1.1 Level 1.1.1 Level Capability Maturity Purpose and Description Measures of Effectiveness 


|) Level 
Integration with Provide feedback to system development 1.8.1 5 Reduce high outyear life cycle cost growth from unplanned training costs to address readiness. Reduce cycle time for curricular review 
Data for human performance issues (Future) Reduced overhead costs to maintain training 
Interoperability 
(ILS/FE&T) 
Incorporation of Support recapitalization of adaptive 1.9.1 Allow for reuse and integration of advanced adaptation to aid in performance improvement without Improve ROI for machine learning initiatives 
Machine Learning learning algorithms refactoring of existing systems. 
Adaptation Support | Integrate advanced sensors 1.9.2 Refine performance information with worksite context and other assessment strategies. Use biometric | Reduce cycle time for curricular review 
identity management. 

Full macro and meta adaptation 1.9.3 Aid learners with course/objective/career planning - optimize learning path Reduce overhead costs to maintain training 
Fully Automated Detailing quota and incentives alerts 1.10.1 Support human capital supply chain management within and across agencies and commands. Optimize cost to maintain given manpower 
Human Capital posture 
Management 
Demand Signals 
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Analytics 
Dashboard 


Other 
Learning 
Tech 


Course 
Le] -|[0)-4 


Level 1 

Investment: 

* Add LRS to decouple 
training records from 
learning technology 

* Common Course Catalog 

Return: 

* Remove vendor lock 

¢ Support basic analytics 

* Support course reuse 


Analytics 
Dashboard 


Other 
Learning 
Tech 


Identity 


Course 
ie] F-|[0)-4 


Level 2 

Investment: 

* Data labeling (globally 
discoverable metadata) and 
governance 

¢ Address globally unique 
identity management and 
privacy 

Return: 

* Support enterprise analytics 

¢ Standardize content curation 


Other 
Learning 
Tech 


Other 
Learning 
Tech 


Activity & 
Resource 
Mgmt 


Course 
Catalog 


Learning 
Event 
Mgmt 


Analytics 
DFT sleley-1ae 


Universal 
Learner 
Record 


Learner 
Profile/ 
CF 


Level 3 

Investment: 

* Migrate to Service Oriented 
Architecture (SOA) 

¢ Establish department-level directory 
services and governance boards 

Return: 

* Synchronized course/content catalog 

* Improve curriculum review cycle 

¢ Shift to CBL, improve user 
throughput 

¢ Enable dynamic federation creation 


Figure 2. CV-3 Capability Phasing. TLA Maturity Model enables phased migration of capabilities. 
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Universal 
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tency Profile/ Record 
System CF 


TLA Core 


Level 4 
Investment: 
* Develop Machine Learning to 
- Improve analytics 
- Automatically generate metadata 
- Perform credential management 
- Perform resource scheduling 
Return: 
* Optimization of learning efficacy 
* Support alignment of job to 
performer 


Edge Ecosystem Core Core Enterprise 


Occupa- 
tional 
Alignment 


Readiness 
Data 


Level 5 

Investment: 

¢ Standardize loLT 
integration specifications 

* Integrate with non-FE&T 
data bases 

Return: 

¢ Lifelong learning planning 

* Human capital supply 

chain integration 
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Table 5. CV-6 Capability to Operational Activities Mapping. References marked “x” are required and “o” are enhanced. 


Operational Activity Decomposition 


Capability ID and Alignment 


Operational 1.4] 1.4 ] 1.2 | 1.2 | 12 ) 1.3 ) 1.3 | 13 | 1.3 | 1.3 | 1.4 | 1.4 | 1.4 | 1.4 | 1.4 | 1.5. | 1.5.) 1.5.) 1.6.) 1.6. | 1.6. | 1.6. | 1.7. | 1.7. | 1.7] 1.8 ] 1.9 | 1.9 | 1.9 | 1.10 
Stakeholder ivi Purpose a2 Net || eel leaealll Asean ollie alll este metomlliee geo cata Soall| ode eae, I ease t ag bere Pee IM Acs lliesdagsl|te Denil Sent ll Pulley alle Sal eet 
Activity 
Learning Command Conduct Assign and collect Learning data xX}|xX}]x}]x}]x}]x]x]xy]x dx xX |X | X X 
Credential Owner Learning Events | Update conferral of credentials based on data x |x] x 
Identity manager Synchronize Maintains globally unique ID tokens X X xX X xX xX xX Xx Xx xX xX xX xX xX X X X X xX X X X X X X 
Credential owner Identity Reconciles credential reporting and digital signatures X X X X 
Learning C d management | "Uses identity handle and ization for Pll 
earning Comman ses identity handle and anonymization for ololololo X 
protection 
Analytics Owner Uses identity handle and anonymization for Pll xlxlxlxilxlixtx X X X 
protection 
Analytics Owner Sharing Establishes data sets for analysis o!|o o!10!o0!/;0/0 O 
Data Model Owner sees Establishes data metamodels for interoperability xilx}i/xixixtxixtxtixtixy]xixtixt)xtixdxtixtdxtxtxtixtxtixdxdtxtixddtx dx 
R 
Learning Command Pa Creates and stores learning records X}xX}xX}]xX}xX}]xX]_xX] x] x] x] xtxitxix it x X 
Competency owner Creating Identifies RCD and association DAG X X X X X X X X x xX | X X 
Warfighting System Competency Identifies tasks, conditions, and standards 
ener Frameworks Oo;}o;0;0};} 0 X X X X X X 
Warfighting Identifies tasks, conditions, and standards X xlxlx x |x 
Requirements Owner 
Warfighting System Manage Validates candidate audience against warfighting tasks 
s X X X X X X 
Owner Job/Duty/Gig 
M&P Owner Definitions Identifies jobs, duties and gigs X X X X X X X X xX | X X 
Analytics Owner Manage Identify new templates and metrics Oo!1o!;lo;o;ro/];o];o;o;o;o];]o;o!;]o};o;}o}]o]}o Oo|;O0O O X xX |X]X}]X |] xX X 
Learning Command Decision Populate and use decision making templates for 
Support Kirkpatrick 1-2 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X 
Readiness Owner Analytics Populate and use decision making templates for 
' . X X X X X X 
Kirkpatrick 3-4 
Policy Owner Maintain Policy | Define policy framework xlxlxtilxtilxlixlxlxlxlxlixtixtixtixtixixtdxtxtxdx dx tx dx X xlxlxixtx X 
Framework 
Content Owner Curate Create, survey and register new learning content X X X X X X X x x Xx Xx Xx 
experiences 
Learning Command Maintain Data Obtain ATC/MOA and use federated data X X X X xX X xX xX xX xX xX X xX X xX xX X X X X X 
DISA Federation Issue ATO/ATC 
Identity manager Enforce non-repudiation, Pll protection and UUID xX}/x}]x}]x}xy}xyx]xy]xy]xyxixixix ix X |X ]}X]X] X X xX |X | X X 
Learning Command Register New Register, use and manage configuration of content xX | X xX/|/x]x]x{txi]tx |x |x X X |X | X 
Data Model Owner Activities Update registry metamodel xlxixtix]xix]txdx x xX | X 
Warfighting Register, use, and manage configuration of content X X x lx X X 
Requirements Owner 
Content owner Migrate legacy | Populate metadata and register content elements xX |x ]x]x {|x xX |X | X X 
Competency owner activities Establish new educational alignment updates x |x] x X 
DISA Maintain Establish cybersecurity policy, Conduct 3PAO xX}/x}x}x}xy}xy]x]xy)xy)xyxixtitxtixtixtixtitxixixtitxtitxtixi x X Xx |X |X ]X |] Xx X 
Learning Command Cybersecurity Establish ATO/ATC x |x x x 
Learning Command Synchronizing Provide for federation of local XI content X}|xX]x}txyxtxyx}txy]xyxitxyxtxyx tx tx X xX | X | X 
Competency owner Enterprise Federation of local educational alignment data x | x x 
Course Catalog : = 
Content owner Populate required metadata for discovery xX | X X 
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Operational Activity Decomposition Capability ID and Alignment 


Stakehold Operational p 1.4] 1.4 ) 1.2 | 1.2 | 12 ) 1.3 ) 1.3 | 13 | 1.3 ] 1.3 | 1.4 | 1.4 | 1.4 | 1.4 | 1.4 | 1.5. | 1.5.) 1.5. ] 1.6. ) 1.6. | 1.6. | 1.6. | 1.7. | 1.7. | 1.7] 18 ] 1.9 | 1.9 | 1.9 | 1.10 
takeholder ivi upeee EV an eee Need ee alee eall ee Mlieeze ni eallipe as IGS 5] ibR2 Nl ee aenl eae ine Gaal diet eeDe Mego ata ego gb aunW cde | allie Deals eed Ol edie P| eS cal ed 
Activity 
Readiness Owner Evaluate Provide MOE to validate weights and completeness of 
X X X X X X 
Competency framework 
Competency owner Frameworks Update Competency Framework (CF) X |X |X |X} X X | X X 
Data Model Owner Update CF metamodel xX |x |x |x X X X X X X X X X X xX |X |X xX | X X 
Analytics Owner Provide data templates to evaluate CF and MOE O X xX |X ]xX |X] xX X 
Competency owner Sharing Share CF X |X | X | X X X X 
Learning Command Competency Use CF X}|xX]xX}]xtx]xti}txyxitx]xitx |x xX |X | X | X X 
: Frameworks ; - 
Credential Owner Archive updates to credentials X X X X X X X X X 
Warfighting System Maintain Data Define data development and sharing policies PDSS x x 
Owner Governance 
Readiness Owner Define data development and sharing policies DRRS X X X X 
M&P Owner Define data development and sharing policies M&P xX |X }xX}]xX }xX }| xX +] XxX X 
Data Model Owner Define data development and sharing policies CF, LP, xlxilxilxtlxtixtixlxtx tx tx X x |x X X x |x X yxlxlxtlxlx X 
Al formats 
DISA Establish privacy and security requirements X}xX}xX}]xX}]xX}]xX xX] xXx] xX] xX] xX?_Xx?txXxtxXxtixitxitxixtixtitxtitxixi x X xX |X |X |X] xX X 
Content owner Develop and implement content data labeling policies xl+xitixtxixitxtix xX |x }]x]x tx X X X X X X X X X X xX |X |X xX | X X 
Content owner Create Create, survey and register new learning content xX xX xX 
Warfighting System Experiences Create, survey and register new reference content sel a o 
Owner 
M&P Owner Review Learner | Identify Detailing and Job assignments x 
Records 
Interoperability Maintain Networking, MOA, ATC and data sharing standards 
Learning Command Device Obtain ATC/MOA and use federated data x | x x |x 
Federations 
DISA Issue ATO/ATC X X X 
M&P Owner Conduct Audits | Verify credentials xX |x ]x}]xX |x X O 
Competency owner Evaluate performance vs requirements xX |x ]x}x]xX O 
Credential Owner Update credential requirements xX |x ]x{]x |x O 
Learning Command Comply with gradebook and transcript requirements, 
: X X X X X O 
conferral review 
Learning Command Evaluate Determine suitability of local content for purpose xX}+xtixtxixitixtix xX O 
Content owner Experiences Perform technology refresh/course update xlxixtixix O 
Data Model Owner Evaluate paradata for data labeling requirements xX |x ]x]x]x O 
Analytics Owner crowdsource analytic templates to evaluators xX |x ]x]x |x O 
M&P Owner Evaluate Personnel detailing, screening, selecting, promotions x |x tx ]x xX O 
Analytics Owner Learners Crowdsource analytic templates to evaluators x}x]x}tx{x O 
Learning Command Review for awarding credential, staff pick up, etc. X X X X xX O 
Learning Command Register New Create and register on site content and activities xX |x }]xXx 
Content owner Activities Create and register globally available content x |x| x 
Learning Command Manage Review evidence and assign credential xX |x ]xX}xX |] xX X |X |X] xX O 
Competency owner Credentials Define qualification standards Oo 
Credential Owner Archive non repudiable copy of credential xXilx]xtix dx X X X X O 
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Table 6. CV-7 Capability to Services Mapping. References marked “x” are required and “o” are enhanced. 


Capability ID and Alignment 


Service Function Decomposition 1.4.1 | 1.1.2 | 1.2.4 | 1.2.2 | 1.2.3 | 1.3.1] 1.3.2 | 1.3.3] 1.3.4] 1.3.5 | 14.1) 14.2 | 1.4.3 | 1.4.4] 1.4.5] 1.5.1] 1.5.2 | 1.5.3 | 1.6.1] 1.6.2 | 1.6.3 | 1.6.4] 1.7.1] 1.7.2 | 1.7.3] 1.8.1] 1.9.1] 1.9.2 | 1.9.3 ] 1.10.1 
Event Schedule event X X X X X X O O O O O O O O O O O O 
Management | Capture event x |x |x }|xyx]x fx tx yxiytx txt x fx txt xyx yx tx }ojyoj}o;o;};o;o};o}o;}ojyoyj}ot}o 
Monitor learner object life cycle X | X | X xX |X | xX | xX | xX | xX | xX | xX |xR]_xX]xX]_xXxyxX;_xX?_ xX }o}o}yo}yo};vo}lmvmOoOUmryW Of} OF} OF} CCF} CO O 
Update event context xAPI messages X X X X X X X X X X X X X X X X X X O O O O O O O O O O O O 
Activity and Federate experience records X X X X X X X X X X X X X X X X X X X X X X O O O O O 
Resource Register content and activities X X X X X X X X X X X X X X X X X X X X X X O O O O O 
Ree Sty CRUD metadata values x}xitxfxfxftxixtxttxfxfdxtxitxtx)xtxdxtdxtxtxtxtxtofofoloflfo 
Provide Enterprise Course Catalog X X X X X X X X X X X X X X X X X X X X X X O O O O O 
Return activity search X X X X X X X X X X X X X X X X X X X X X X O O O O O 
Competency | Search competency X X X X X X X X X X O O O O O 
Management | Update/fetch career state xX xX xX xX x xX xX xX x xX O oO O O O 
CRUD/Import job pool X X X X X X X X X X O O O O O 
CRUD/Import competency networks X X X X X X X X X X O O O O O 
Calculate change in learner competency state X X X X X X X X X X O O O O O 
Manage learner credentials X X X X X X X X X X O O O O O 
Mange leaner records X X X X X X X X X X O O O O O 
Import readiness data X X X O O O O O 
Identity Mange identity groups O O O O O O O O O O O O O O O O O O O O 
Management | CRUD/fetch users in enclave He Be ae ae ae ae re a oe Nh. A ae | a we) ae) Oe free) Oe | Oe else |) ee We da See | ae 
Manage user personas X X X X X X X X X X O O O O O O O O O O O O O O O O O O O O 
Manage user digital records X xX xX X X xX X xX xX X O O O O O O O O O O O O O O oO O O O O O 
Virtualization | Manage dynamic endpoints xX xX xX X xX xX xX xX xX xX xX xX xX xX xX xX xX x xX x xX x xX xX xX x xX xX xX x 
Management | Manage dynamic resources ya ee ae | | | en D> | ea ce ee eT (a te ce a ce ee X 
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3.0 DATA AND INFORMATION Views (DIV) 
3.1 DIV-1 Conceptual Data Model 


The DIV -1 describes the types of operational information by data owner class. The headings in the 
following section represent classes of interfaces shown on the OV-2 in Section 3. 


3.1.1 Governance Procedures 


The future learning ecosystem is a living, constantly evolving enterprise of products and processes 
where the scope and complexity of tools and data involved require policy and procedures for 
governance’. The total scope of content, user base, and record retention requirements for the DoD is 
too large for a single data store or suite of tools to provide utility to all users at acceptable performance. 
This requires a federated systems approach, which necessitates continuous coordination, reviews and 
updates. 


Data must be maintained at the appropriate classification level, but also be available to push from low 
to high enclaves for decision makers to get complete pictures of the status of personnel, curricula, 
materials, and readiness. Governance procedures providing data structure compatibility, message 
interoperability, coordination between enclaves, and continual review and update of data mappings are 
necessary for success in this environment. 


A critical part of this solution is the configuration of data elements and service deployments over time. 
This requires a robust governance for federated data and the consistent migration of legacy content and 
data to the new stores. Governance must exist at several levels (enclave/organizational, federated 
integrated product team, and the broader ADL community), and the appropriate level of participation, 
and frequency for each governance board needs to be identified. For these purposes, organizations are 
asserted to own (govern) the types of information presented below. 


3.1.2. Education and Training Policy Owners 


The TLA policy framework defining the ecology of the objective end-state is developed by the ADL 
Initiative, under the Force Training and Education Branch and reporting through the Under Secretary of 
Defense for Personnel and Readiness - OUSD(P&R). The TLA is composed of commercial standards and 
technical specifications developed under the TLA research portfolio at the ADL Initiative. 


3.1.3. The TLA Policy Framework 


The TLA policy framework includes the experience API (xAPI), currently undergoing revision by the IEEE 
Learning Technology Standards Committee (LTSC) to version 2.0. It will transition to the IEEE 9274 
Standard. The Reusable Competency Definition Object (RCD) is also under review, to create an update to 
IEEE 1484.20.1. Additional TLA specifications are being evaluated during 2019 and 2020 with a target 
date for Initial Operational Capability (IOC) of 2021. Other elements that require governance include: 


e DODI 1322.26 (Reference A) — policy guidance managed by the ADL Initiative 
e Business Logic — rules for being TLA compliant 


5 Walcutt, J.J. & Schatz, S. (Eds.) (2019). Modernizing Learning: Building the Future Learning Ecosystem. 
Washington, DC: Government Publishing Office. License: Creative Commons Attribution CC BY 4.0 IGO. 
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e  TLA Specification — references the xAPI specification and provides a management profile for 
ensuring consistent state changes between TLA core systems 

e The TLA Master Object Model (MOM) — composed of the reserved verbs listed in the SvcV-10b view 

e Federation Negotiation Process — rules for registering components, profiles, and content and 
ensuring federated identity management, semantic alignment of metadata, namespace 
management, and alignment of activity-content-competency object handles 

e Industry governing bodies for constituent specifications 
o Dublin Core™ - Learning Resource Metadata Initiative (LRMI) and LOM 2.0 

IMS Global™ - Learning Tools Interoperability (LT1) 

IMS Global™ - OpenBadge3.0 

Open API Initiative™ - Open API 

Open ID Foundation™ - Open Identification Connect (OIDC) 

Credential Engine™ - Credential Transparency Description Language (CTDL) 


oO 0 0 0 


3.1.4 Competency Based Talent Management 


Governance considerations include: 


e Assignment of competency owners and lifecycle management process 

e Migration of legacy curricula and standards to populate the initial Competency Frameworks 

e Governance for maintenance of the Competency Frameworks over time resulting from changes to 
Warfighting Systems 

e Policy guidance to manage condition and time-based skill decay and de-credentialing policies 

e Credential reporting 


3.1.5 Social Learning Policies 


e Instrumentation of social networks and tools 
e Standardization of tools for operational security and stability 
e Standardization of tagging and linking to align with xAPI profiles 


3.1.6 Data Model Owners 


e Data Metamodels 

Configuration of Competency Framework metamodel 

Human Performance Requirements Model 

Content and Resource Metamodel definition 

Enterprise Learner Profile Metamodel 

xAPI profile development for edge systems (verbs, object life cycles, and learning technology) 


oO 0 0 0 


e Metadata Name-Value-Pair definitions 
o Local, regional and global attributes 
o Maintenance of namespaces for semantically equivalent terms 


3.1.7. Cybersecurity (DISA), Identity Management, and Data Federations Policy 


A basic feature of any networked system is its cybersecurity posture, including privacy, non-repudiation, 
integrity, security and reliability. In the future learning ecosystem, non-repudiation requires federated 
ID management. Privacy and security require anonymization tokens for user data to prevent 
accumulation of Pll or aggregation of exploitable data that can present a security risk. 
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In classified domains, this is complicated by the use of multiple security enclaves. Currently, the Defe 
Manpower Data Center and Joint Interoperability Test Center have equities in their management of 


nse 


tokens and protocols, as well as open industry standards and best practices, so this is likely to change 


and mature as the ecosystem grow. 


e Universal Unique Identifier (UUID) and PII protection, including technology used and clean room 
server policies 
e Authority to Operate (ATO) Memorandum of Agreement (MOA)/ Authority to Connect (ATC) 
o Multi-level security Cross Domain Solution 
o Network and server virtualization and dynamic endpoint management 
o Cybersecurity and Global Information Grid integration - DODI 8500 (series) 
o Maintaining integrity and searchability of federated data structures (i.e. Enterprise Course 
Catalog) 


o Leveraging of virtualized identity management (central logins), networking and file management 


services as technology changes (Federated ID) 
o Encryption, tokens, and digital signatures 
e Asynchronous Data Management (for deployed units or individuals) 
o Time stamping 
o Asynchronous data updates 


3.1.8 Learning Commands (Schoolhouses, Instrumented Field Activities) 


Learning commands represent the fundamental building blocks of the ecosystem and have their own 
equities in locally relevant user data, collection and archival requirements. Moreover, they have 
instructional goals that are enhanced by analysis of learner data archived elsewhere in the DoD ina 
different part of the learner’s trajectory, so they are most concerned with data federation and data 
labeling policies. 


e Federated LRS data - Learner Experience Records and evidentiary chain supporting credentials 
e Federated Experience Index data - Enterprise Course Catalog information 

e Federated Learner Profile data - Learner preferences, competency and credential history 

e Local Metadata, User data and data labeling procedures 

e Namespace management and coordination 


3.1.9 Competency Owners (Rating, MOS, NEC, AFSC, CoE, TCM) 


The TLA concept recommends the decomposition of curriculum into competencies and content. The 
legacy curriculum owners are likely to absorb one or the other of those new roles. In the case of 
tradecraft, the competency owner may the Program Management Office (PMO). Competency owner 
define the job roles, and the nature of the KSAOs, and competency maps for the given level of maste 
that define the job requirements. 


e Federated Competency Framework data 

o Competency Framework information 

o Warfighting system and warfighting requirements 
e DoD schema management 
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3.1.10 Content Owners (Acquisition Agencies, Local Education and Training Facilities) 


Content owners are responsible for the creation, acquisition, search, evaluation, assessment, validation 
and maintenance of the activities and content used to provide learning experiences. Part of content 
curation process is the configuration management and status accounting of normative references for 
the performance standards. 


e Curation versus content development acquisition policies 
e = Activity/content registration policies 

o Content management review, approval, and audit 

oO Grain Size normalization 


3.1.11 Analytics Owners (e.g. OPA) 


It will be valuable to crowd source and share analytics templates and business logic amongst the various 
learning commands and curricula maintainers. Currently, the Office of Personnel Analysis (OPA) has 
some equities in analytics, but this will likely mature as the ecosystem grows. 


e “Crowd sourcing” of innovative analytics /decision support templates 
3.1.12 Manpower and Personnel Owners (e.g. Detailers) 


Human Resources or M&P commands have interest in TLA data because they define job and manning 
requirements, detail personnel based on their competencies and career trajectories, and manage the 
evaluations which in the legacy solution are the only place competencies other than Knowledge, Skills 
and Abilities (KSA’s) are usually maintained (i.e. in performance evaluations). Eventually M&P can be 
fully integrated with feedback and feedforward demand signals to the TLA stocking the human capital 
supply chain. 


e Job/duty/gig definitions 
e Configuration management 
e Demand signals 


3.2 DIV-2 Logical Data Model 


Figures 3 & 4 show the logical relationship of concepts used throughout the TLA. The colored icons show 
the notional allocation of these data elements to core data services described in the main report. Table 
6 is the data dictionary for this model. 


3.3 DIV-3 Physical Data Model 


Figure 5 shows the class structure of the experience API (xAPI), IEEE P9274. These models will be 
extended as the TLA policy framework and specification tree matures. 
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e Must have consistent style 6 : 
Content_Set Learner Profile 


Handle:URI e Requires synchronization with external data ContentList[URI(ContentMetadata),sequence][] Ce Competency Framework 
Content:URL[] source Author:URI(Person) 


«Purpose Learning Experience 


s 
equences IsCourse:BOOL @ Index 


Description:$ 
Bookmark:URL[] 


ats cl 19 
Activity_Metadata Language:RFC5646 eb) _ReferencelD:s a @ LRS 
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Handle:URI Metadata:NVPS[] : ViewCommonCourseCatalog() : 
ees ConifigurationHistory:ConfigurationRecordl] Importance:INT a = Conformance Suite 
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AdjudicationAuthority:URI(Person | Interest Group) SignatureAuth:URI(Person| Interest Group) pare SEN eines se _CandidateAudience:BOOL 
ContentAllowed:Content_Metadata|] RequiredAptitude:Sff} == | nentnnenn mene nnnn sl ies biel areata i tated UpdateMembership() 


Learning Exercise 


GetReadinessData() 


QuotaType:enum 
Address:URI 
Weighting:Float 


Metadata:NVPS[] 


HandleLearningAlert() 
KSAO:KSAOEnum ! 


Handle:URI 


Forms 


: ‘ : Origin:URI j 
ConfigurationHistory:ConfigurationRecord[] EducationalAlignment:RCDI{] Version:INT SoM ee Sane 
"Reaeniieradacat) aaah ck cam ec Context:URI(ActivityMetadataHandle) senneenennnennnnneeenneceeeneneansnaecsnnnaeenseaeeneenonennceen : 
PnCinc Nae oni Resources:URIContentMetadataHandle[] SetGoal(CheckAbility()) Alias:S 
: ; 5 src LearningActivityLaunch() EvaluateEvidence() Handle:URI UUID:$ 
ANISM ale EvaluateTrust() Name:$ Z 
CreateCurriculum():ContentSet UpdateParadata() iti Handle:URI 
: lid UpdateCareerState() Condition: Aptitude:NVPS 
ActivitySearch() ValidateResources() : ptitude: [] 
LearningRecordRequest() ; 
EventRequest() Rene PersonaRole:Role_Persona|[] 
A eqCompetency() F UXRole:UXRoleEnum{] 
copaltes Updatelnference() Job_Duty_Gig ConfigHistory:ConfigurationRecord[] 
UpdateTaxonomy() 9 


CareerTrajectory:CareerTrajectory 


Learning_Event plas CompetencyState:Competency[] 
RCD_Association ; LearnerState: MOMLifeCycleVerbEnum 
Paradata:xAP! = JobCode:S ‘ 
: Graphs LearnerPreferenceAttributes:NVPS[] 
LearningRecord:xAPI[]} | Authority:URI{(Person|Interest Group) 4 : ‘ 

a QED:BOOL Arranges: ‘ CredentialState:Credential[] 
ActivityStatement:xAPI a IsMilestone:BOOL : ; ' 
DrivneyLeuehinT _ Handle:URI(RCD| Standard | Condition) 7th Arranges RequiredSkills:URI(Competency)[] Goal:RCD|Competency |Job_Duty_Gig| Credential[] 

¥ : _ Source:URI(RCD | Standard | Condition) tn : ‘ DS ANA TaskList:LearnerTask[] 
StandardType:StdType_enum a RequiredAptitudeURI(Person::Aptitude)[] «| wannnnnnnnccnnnnnncnnennennnennnsnnnnanananennnennnnnnannnnnnnnnnnsannses 
exogenn peceeee dee eee errs _ Target:URI(RCD| Standard | Condition) ManningPlanCode:${] GetLearningPath() 
pacer ely Ap!) _ Weighting:Float Pchecieisauchain = UpdateCareerState() 
earnerstate ate|x. 
SQN:BOOL UpdateCompetencyState 
ReadLearningRecord() Competency ImportJob() p ¢ y () 


5 GoalSelect() 
LearningRecordRequest|[] 


wee Requi PropogateRolesandPermissions 
OlCSApproval(BOOL) Haleeeh ota ee epee suet 0) 
asteryLevel: 


<—____ 
A C Competency_Framewor i 
Constrains Credential Dinente Handle:URI P v— ‘ en 
. F ! y 
| TaskMetadata:NVPS[] Authority:URI(Person|Interest Group) HandleLearningAlert() 


RequiredAptitude:NVPS[] 


xAPI_Profile AwardAuthority:URI(Person) eee Organizes ConfigHistory:ConfigurationRecord[] 
EffectiveDate:ISO8601 J aera Loterestere He} _Description'$ = ttees 
Purpose:S | Status:CredStatusEnum May Represent ilest -BOOL MaintainConfiguration() Implements 
Owner:URI(Person|InterestGroup) OccupationalStandard:OCSTD[] a ehile tone: wannnnnn naan nnnnnnnnnnnnn anna annnnnnnns CF_Update(:RCD[], RCD_Association[]) 
ObjectLifeCycle:S[] QualificationStandard:URI(Content) CFSearch Role_Persona 
Extensions:NVPS[] Handle:URI AlertidGroupToChange() 
Namespace:URL Supports Description:S AlignOCsTD() Description:$ 
Badge:OpenBadge MakeAssertion() Authority:URI(Person) 
IsMilestone:BOOL Handle:URI 
ParadataContextSet ““issueDigitalBadge() SSCS RequiredAptitude:NVPS 
ManageConferral() Alias:EMailAddr 
Handle:URI VerifyCurrency() ; Prerequisites:URI(RCD)[] 
Context:NVPS[] Composes (Recursively) 


Provides Evidence for- 


Figure 3. DIV-2 Logical Data Model. 
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<<Enumeration>> 


<<Enumeration>> <<Enumeration>> 


UXRoleEnum BYoyaar-lial=ave lan) MOMLIifeCycleVerbEnum 
Administrator Cognitive Recruited 
Learner Affective Assessed 
Observer/Instructor/Controller/ Psychomotor- Fine Motor Detailed 
Supervisor Psychomotor- Gross Motor Mobilized 
Curriculum manager Environmental Constraint Employed 
Competency manager Metacognitive Selected/Screened 

Social Promoted/Demoted 
Motivational Transitioned 
<<Enumeration>> ReleasedPlanned 
Augmented 
KSAOEnum <<Enumeration>> 48 
Directed 
Knowledge StandardTypeEnum Requested 
Skill Approved/Denied 
Ability MOP Explored 
Other MOE Clarified 
Behavior Captured 
<<ComplexType>> Assessed 
Asserted 
<<Enumeration>> CompetencyState Verified 
QuotaTypeEnum SkillSet:RCD[] Valiieted 
i Located 
Achievements:Competency[] inforrad 
_ Agency Specific... IsCurrent:BOOL SS 
oe Conferred 
IsPending:BOOL Qualified 
Ee enMmeradonse IsSuspended:BOOL Certified 
IsRevoked:BOOL Soriaived 
AttachmentEnum EffectiveDate:ISO8601 ees ‘i 
Reni 
PendingDate:ISO8601 : Boe 4 
Agency Specific — roritize 
Recommended 
<<Enumeration>> Curated 
DataTypeEnum Scheduled 
Regulated 
<<Enumeration>> Integer Projected 
ContextEnum Double Contextualized 
Float Evaluated 
Agency and Program Specific IRI Surveyed 
ISO8601 Tracked 
String Projected 
<<Enumeration>> Char waived 
ResultEnum BOOL launched 
RFC 5646 initialized 
MediaTypeSpecific passed 
failed 
<<ComplexType>> satisfied 
<<ComplexType>> Devi completed 
Occupational Standard (OCSTD) exc abandoned 
terminated 
DeviceName:$ 


IPAddress:IPv4 


<<Enumeration>> 


VerbClassEnum 
(Cognitive/Psychomotor) 


Find 
Identify 
Label 
List 
Match 
Name 
Assess 
Change 
Chart 
Choose 
Compute 
Construct 


(ie 
He 


i 


enetenea| | 


e 
3 


<<Enumeration>> 


VerbClassEnum 
(Affective/Metacognitive) 


ACH-Achievement Orientation 
CO-Concern for order, quality and 
accuracy 

INT-Initiative 

INFO-information seeking 

IU Interpersonal understanding 
CSO- customer service orientation 
IMP impact and influence 

OA- organizational awareness 

RB relationship building 

DEV developing others 

DIR directiveness, assertiveness and use 
of positional power 

TW teamwork and cooperation 

TL Team leadership 

AT analytic thinking 

CT conceptual thinking 

EXP technical professional managerial 
expertise 

CST self control 

SCF self confidence 

FLX flexibility 

OC organizational commitment 


<<ComplexType>> 


Taxon 


Framework:$ 
EnumeratedLevel:Int[] 
Enumerated List:S[] 


<<ComplexType>> 


NameValuePairSet (NVPS) 


AttributeName:URI 
AttributeDataType:DataTypeEnum 
AttributeValue: DTC 
AttributeDescription:S 

Scope: 

TimeStamp:ISO8601 


<<ComplexType>> 


XAPI 


Actor:URI (Person:Handle) 
Verb:URI(Profile:Handle) 
Object:URI (Profile:Handle) 
Timestamp:ISO8601 
Stored:ISO8601 

Language: RFC5646 
Result:ResultEnum 
Context:ContextEnum 


Authority:URI(Person(handle) |Interestgro 


up(Handle)|ActivityMetadata(handle)_ 
Attachments:AttachmentEnum[] 
Version:S 


<<Enumeration>> 


ContentType 


ePublication 

Publication 

Digital Video 

Video 

SCORM Course 

LMS - Assessment 
Assessment - Other 

ITS 

CMS Material 
SocialMediaSharedFile 

LVCG Simulation Scenario file 
IETM/EPSS 

Trainee Guide 

OICS Comment Sheet/Checklist 
Game Level 

ERP 


<<ComplexType>> 


ConfigurationRecord 


SequencelD:LONG 

TimeOfChange:ISO 8601 
AuthorityForChange:URI(IdentityGroup 
| Person::Handle) 

RecordType:$ 

AttributeChange:NVPS[] 


ReferenceDocument:URI(Content) 
InDocumentReference:$ | 
Type:S 
Interface:URL 
MACAddress:S 
DYoyaar-]ia) PhysicalLocation:S 
Certificate:FIPS140.2 


<<ComplexType>> 


DomainEnum:DomainEnum 
Level:INT {1-7] 


Figure 4. Complex Data Types and Enumerated lists for DIV-2 Data Model. 


<<ComplexType>> 


LearnerTask 


<<ComplexType>> 


CareerTrajectory 


State:xAPI(MOMLIfeCycleverbs){] 
JobList:Job_Duty_Gig[] 
CareerEndpoint:Job_Duty_Gig{] 
Classifier:S[] 
CurrentGoals:URI(Competency)[] 


ScheduledEvent:URI(LearningExperience:Handle) 


ScheduledTime:ISO8601 
SuspenceTime:ISO8601 
Authorized:URI(Person:Handle) 
Assigner:URI(Person:Handle) 


<<Enumeration>> 


CredStatusEnum 


Current 
PendingReview 
OutOfCurrency 
Suspended 
Revoked 


<<ComplexType>> 


OpenBadge 


ContextJSONLD 
ID:JSONLD 
Type:JSONLD 
Recipient:JSONLD 
IssuedOn:JSONLD 
Verification:JSONLD 
Badge:JSONLD 
Issuer: JSONLD 
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Table 7. DIV-2 Data Dictionary. 


Activity_Metadata 


Attributes 
Address 


Data Type 
URI 


Definition 


RESTful location of the content metadata mapping within an 
Experience Index 


Activity_Metadata 


AdjudicationAuthority 


URI(Person) Array 


Trusted agents that can positively or negatively adjudicate learner 
success at a scenario, exercise or activity 


Activity_Metadata 


Authority 


URI (Person) 


The author or registrant of the activity 


Activity_Metadata 


ApprovalAuthority 


URI (Person| Identity Group) 
Array 


Authority to select the content to satisfy a competency element or 
curriculum 


Activity_Metadata 
Activity_Metadata 


Bookmark 


ConfigurationHistory 


URL 


Configuration Record Array 


Location within the content 


List of content or competency attributes which have been changed 
over time in this association 


Activity_Metadata 


Description 


String 


Purpose of the learning activity 


Activity_Metadata EstimatedTime ISO8601 Mean time to complete the activity 

Activity_Metadata Handle URI Internal reference for the learning activity 

Activity_Metadata Device Device Array Complex type for registered devices and clients as appropriate 
Activity_Metadata Metadata NVPS Data that describes the content from metadata standard 
Activity_Metadata QuotaType Enum How cost of attendance at the experience or content is remunerated 


Activity_Metadata 


Required Resources 


NVPS 


Consumables, instructors, classrooms, computational resources, 
laboratories or other materials necessary for the experience 


Activity_Metadata 


SchedulingAuthority 


URI (person or interest group 
object handle) 


Person or interest group authorized to schedule the content or act as 
registrar 


Activity_Metadata 


Weighting 


Float 


Contribution of the activity towards demonstrating competence 


Activity_Metadata 


ContentAllowed 


URI (Content_Metadata) (Array) 


Content that can be used with the specified activity 


Competency IsMilestone BOOL Signifies whether competency establishes a personnel milestone (for 
planning learning trajectories). 

Competency Masterylevel INT Level of proficiency required for a competency as defined for use ina 
job/duty/gig 

Competency Name_Description String Short reference handle for the competency (as a map of competency 


objects, level of mastery, associated standards and conditions or 
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Class Attributes Data Type Definition 
contexts, and the relationships between them described as a DAG for 
defining the competencies 
Competency RequiredAptitude NVPS String List of user aptitude attributes as prerequisite to attempt 
achievement of the overall competency 
Competency CnCmetadata NVPS String Metadata associated with task (used to scrape for applicable content) 
Competency Handle URI Internal handle for referencing the competency network 
Competency Authority URI (person object handle) Person or organization that owns the competency element 
Competency PrivacyLevel Integer A scaled level of privacy for the evidence provided by the record. 


Competency_Framework 


ConfigurationHistory 


ConfigurationRecord Array 


The ordered list of changed attribute values, authorization and date 
of change from complex data type 


Competency_Framework | Authority URI (Person or Identity Group The person or organization who “owns” the Competency Framework 
object handle) definition 

Competency_Framework |Description String A short title for describing the Competency Framework (may tie to 
organization, MOS, Rating, or some subset) 

Content_Metadata Handle URI Internal object handle for referencing the standard 

Content_Metadata Content URL Array An array of possible locations the content can be located at 

Content_Metadata Description String A description of the piece of content 

Content_Metadata Bookmark URL Array A pointer to the content’s location on the internet 

Content_Metadata Language RFC5646 The language(s) needed to use the content 

Content_Metadata MediaType ContentTypeEnum An enumeration of the kind of content the content is, such as ebook, 
pdf, or movie 

Content_Metadata Metadata NVPS Data that describe the content, according to the LRMI, TLA, and local 


extensions for metadata 


Content_Metadata 


ConfigurationHistory 


ConfigurationRecord Array 


List of content or competency attributes which have been changed 
over time in this association 


Content_Metadata 


Authority 


URI (Person | Identity Group) 


The person(s) allowed to assign and edit the content 


Content_Metadata 


NormativeRef 


BOOL 


Whether the content is a normative reference for the listed 
competencies (i.e. tech manual describing a skill) 
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Class Attributes Data Type Definition 

Content_Metadata DocumentNumber String The record number used to reference the document, especially if a 
normative reference (e.g. AFM65-10, MCWP 5.0. NATOPS 80R14) 

Content_Metadata Version Integer x.x.X version number of the piece of content 

Content_Set Author URI (Person object handle) Instructor or course manager who approved course 

Content_Set Purpose String Catalog entry or description of course, or other purpose for the set 

Content_Set IsCourse BOOL Used to index records for Enterprise Course Catalog searches 

Content_Set ReferencelD String The agency specific record number for the course listing 

Content_Set ContentList URI (Array) An array of the content object handles in the content set 

Credential IsMilestone BOOL Signifies whether competency establishes a personnel milestone (for 
planning learning trajectories). 

Credential AwardAuthority URI(Person) Who signs off as the issuing or updating official 

Credential EffectiveDate ISO8601 What is the status date effective by 

Credential Status CredStatusEnum A status on awarding the credential 

Credential Badge OpenBadge2 Definition of the digitally signed credential in the OpenBadge 2.x 
standard 

Credential QualificationStandard URI (ActivityMetadata object The normative reference, document or instruction (e.g. Field Manual, 

handle) Technical Instruction) that specifies the need 

Credential OccupationalStandard OCSTD array The Occupational Classification Standard (OCSTD) from O*NET that 
defines the framework 

Credential Handle URI Internal handle for referencing the competency network 

Credential Authority URI (person object Handle) Person or organization that owns the competency element 


Interest_Group 


CandidateAudience 


BOOL 


The group of identified users is assigned to collectively assign a 
training requirement (e.g., a class) 


Interest_Group Collective Address URI Name for referring to the collection of entities (e.g. All Pacific 
commands, section 12 of the 2021 fire controlman class). Typically 
for classes, it is class number based on year and number of classes 
being taught. 

Interest_Group IsClassSession BOOL Used to filter faster for classes and sections 
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Class 


Interest_Group 


Attributes 


Member 


Data Type 
URI Array 


Definition 


Internal handle or handles used for identifying humans logged into 
the local instance of the system (used to protect PII) 


Interest_Group 


PersonaRole 


URI (Persona_role) 


Persona role of the learner or group of learners 


Interest_Group Protected BOOL Interest group membership changes must be approved by an 
observer, instructor, controller, or supervisor 

Job_Duty_Gig Authority URI (Person or Interest Group Curriculum or competency definition authority 

object handle) 

Job_Duty_Gig JobCode String Branch specific occupational code (e.g., Naval Enlisted Classification, 
Military Occupational Specification, Air Force Specialty Code) 

Job_Duty_Gig Name String Short name for referring to the job, duty or gig (e.g., imagery analyst, 
collections) 

Job_Duty_Gig RequiredAptitude String array Aptitude values (i.e. from ASVAB or AFQT) required to perform job 

Job_Duty_Gig RequiredSkills URI (Ability object handle array) |Pre-requisite abilities to pursue the job 

Job_Duty_Gig IsMilestone BOOL Job establishes a personnel milestone for planning learning 
trajectories 

Job_Duty_Gig Handle URI Internal handle for referring to the job 

Job_Duty_Gig ManningPlanCode String Array Reference number or code in the governing M&P document that 


describes or justifies the position 


Learning Event 


ActivityStatement 


XAPI 


Atomic level learning event where leaner did something, where 
“something” includes review of some type of content or conducting 
some job experience, as per xAPI standard used as evidence of 
competence 


Learning Event 


Experience 


xAPI Array 


A linkage of activity statements that must be taken together to 
constitute “evidence”. The verb of the xAPI statement, used to 
explain how the user interacted with content to learn 


Learning Event 


Paradata 


xAPI 


Proforma record of the context of the learning experience, including 
cognitive and environmental effects 


Learning Event 


StandardType 


StdType_enum 


An enumerated data type of the type of evidence (MOP or MOE) that 
the record defines. MOP come from learning technology; MOE 
typically come from operational data sets 
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Attributes 


Data Type 


Definition 


Learning Event 


PrivacyLevel 


Integer 


A scaled level of privacy for the evidence provided by the record. 


Learning Experience 


EducationalAlignment 


RCD (Array) 


List of competencies the activity can satisfy or enhance 


Learning Experience Context URI (ActivityMetadataHandle) The player, reader or environment used to conduct the learning 
exercise 

Learning Experience Resources URI (ContentMetadataHandle) |The files or other resources required to conduct the exercise 

Paradata_Context_Set Handle URI Internal handle for referring to the paradata attribute set 

Paradata_Context_Set Context NVPS Array List of potential attributes that can be reported by the activity 
provider to define the paradata 

Person Ability URI (ability object handle) Array |The learner’s abilities (defined from ability classes) 

Person Aptitude NVPS Array The learner’s aptitudes (defined from aptitude classes) 

Person Handle URI Anonymized internal reference for a person (used as “actor” in xAPI) 

Person CompetencyState Competency Array An array of all the competencies the person has had asserted and 
verified 

Person CredentialState Credential Array An array of all of the competencies the person has had certified and 
conferred 

Person LearnerState MOMLifeCycleVerbEnum The current learner state as managed by the learning event manager 

Person LearnerPreferenceAttrib |NVPS Array Attributes of the learner used for adaptation decisions or algorithms 

utes (reference installation specific) 
Person PersonaRole URI (Role_Persona object Personas or roles of the person (e.g. Sailor, division officer, watch 
handle) Array stander, analyst) 
Person UUID String Externally valid reference for person (CAC or other UUID) 
Person Goal Job_duty_gig, RCD, Credential or | Lists the fully recursive array of competency objects that are 
Competency Array currently pursued by the learner. May be arbitrarily deep and broad 

Person Career Trajectory CareerTrajectory An array of jobs that define the past, present and candidate future 
jobs for the learner on their current trajectory 

Person ConfigurationHistory ConfigurationRecord Array The ordered list of changed attribute values, authorization and date 
of change from complex data type 

Person TaskList Learner Task Array The list of tasks that have been formally assigned as complex types 
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Class Attributes Data Type Definition 

RCD BehaviorDomain DomainEnum Whether competency element is cognitive, psychomotor, affective, 
metacognitive, social or motivational 

RCD Importance INT Weighted requirement for career progression 

RCD KSAO KSAOEnum Identifies whether competency element is Knowledge, Skills, Ability, 
Behavior, or other 

RCD Metadata NVPS string An array of one or more metadata elements that describes the 
competency object 

RCD Needed at Entry BOOL Is competency needed at entry for the job/gig trajectory (part of 
learning validation logic for pre-requisite skills or experiences) 

RCD RequiredAptitude String Array List of user aptitude attributes as prerequisite to attempt 
achievement of the competency 

RCD SignatureAuth URI (Person or Interest Group Trusted agents that can assert competence from evidence 

object handle) 

RCD Task String Task, behavior, or measurable elements (in the case of knowledge 
competencies) that are demonstrated by the competency 

RCD TaskMetadata NVPS An array of one or more metadata elements about the tasks that are 
included as part of a competency 

RCD Handle URI A Uniform Resource Identifier (URI), a string of characters that 
unambiguously identifies a resource 

RCD Origin URI The URI of the normative reference (content metadata) that specifies 
the competency requirement 

RCD Verb String The verb that defines the task statement for the RCD if it is sourced 
from an existing learning objective or stated as a task. 

RCD VerbClass VerbClassEnum Generic category of the task (e.g., operate equipment is “perform”) 

RCD VerbNamespace String Community of practice whose definition of the verb applies 

RCD LearningModelLevel Taxon Array Defines which learning model framework and level the verb applies 
(e.g. Bloom, Merrill) 

RCD BehaviorDomain Domain array The skill component and level from 1-7 of the behavior 

RCD Version INT Version of the competency object definition 
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Attributes 


Data Type 


Definition 


RCD_association Handle URI A reference for the vector map represented by the sequence of 
associations 

RCD_association QED BOOL Quod Erat Demonstrandum; Association that cascades positively 
downwards (if high level competency is asserted, lower level is 
automatically asserted) 

RCD_association Source URI (of RCD, Standard, or Source URI of competency object that is upstream on the association 

Condition) 

RCD_association SQN BOOL Sina Qua Non; Association that cascades negatively upwards (if low 
level competency is de-credentialled, the higher level is automatically 
de-credentialled) 

RCD_association Target URI (of RCD, Standard, or Target URI of competency object that is downstream on the 

Condition association 

RCD_association Weighting Float Covariance weight or contribution of the downstream competency 
object to the upstream object 

Required_Context Alias String Mapping for “condition” if specified in the associated Competency 
Framework more concretely or explicitly (e.g. context, environment, 
organizational level) 

Required_Context Condition String A condition under which the definition of competency is appropriate 

Required_Context Handle URI Internal object handle for referencing the condition (used in xAPI 
extensions) 

Required_Context Name String Screen name for describing the condition 

Role_Persona Alias EMailAddr Internal reference handle provides a name that cannot traced back to 
Pil 

Role_Persona Authority URI (person object Handle) Curriculum or professional standardization authority 

Role_Persona Description String Short description of the purpose or scope of the job/duty/gig 

Role_Persona Handle URI Local code or nomenclature for the job/duty/gig as appropriate (e.g., 


billet code), which resolves to local database and handles 


Role_Persona 


Prerequisite 


URI (RDC object handle) Array 


Required competencies to perform the job 


Role_Persona 


RequiredAptitude 


NVPS 


Required Aptitude of the person to perform the job 
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Class Attributes Data Type Definition 

Standard Alias String Way “standard” is specified in the associated Competency 
Framework (e.g. level of proficiency) 

Standard Criterion Integer The level of performance that defines the standard 

Standard Handle URI Internal object handle for referencing the standard 

Standard Measure Activity Array The objective measurement for establishing the standard 

Standard Name String Screen name for describing the standard 

Standard NormativeRef URI (ActivityMetadata) Array Identifies the normative reference defining the standard within the 
content library (e.g., tech manual that describes a procedural skill) 

Standard Standard Type Standard TypeEnum Whether the standard is a MOP for a competency object (means of 
asserting competency) or an MOE for an entire competency or range 
of competencies (i.e. organizational or outcome-based performance 
metric against which to validate individual performance and 
instructional efficacy) 

xAPI Profile Purpose String Type of learning activity or data sources that would use this profile 
(e.g. eReader, Personnel Database) 

xAPI Profile Owner URI (Person or Interest group Creator of the profile 

object handle) 

xAPI Profile ObjectLifeCycle String Array List of allowable xAPI verbs within the Profile 

xAPI Profile Extensions NVPS Array List of attribute and enumerations or string masks for extensions, 
results and attachments 

xAPI Profile Namespace URL Globally unique way of referring to elements within the profile 


Class 


BehaviorDomain 


Attributes 


DomainEnum 


Complex Data Types (depicted in Figure 4) 


Data Type 


DomainEnum 


Definition 


Cognitive components 


BehaviorDomain 


Level 


Integer 


Level of cognitive components from 1-7 


CareerTrajectory 


CareerTrajectory 


State 


Job List 


xAPI Array 


JobDutyGig Array 


Used manpower verbs from enumerated data type 
LearningEventLifeCycleVerb 


List of jobs the learner has held 
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CareerTrajectory CareerEndpoint JobDutyGig Array The list of jobs that the learner wants to hold on their current career 
trajectory — with branches and options 


CareerTrajectory Classifier String Array The learner’s current career classification (e.g., MOS/NEC/specialty 
code) 
CareerTrajectory CurrentGoals Competency Array or RCD Object | Competency objects the learner is currently pursuing or has been 
Array assigned by the observer, instructor, controller, or supervisor 
CompetencyState Skillset RCD Array All the lower level items, especially those not belonging to core 


framework, that the individual has demonstrated competency 


CompetencyState Achievements Competency Array All the complete competencies, at a given level of mastery, that 
represent a graph of RCD, that the learner has mastered 


CompetencyState IsCurrent BOOL For those competencies that require periodic demonstration 
observed by a designated person that the learner has maintained 
currency in the task 


CompetencyState IsPending BOOL A Boolean depicting if the competency state needs OICS approval 
before becoming official 


CompetencyState IsSuspsended BOOL The learner has been administratively removed from being 
considered competent (e.g., medical hold, out of currency) 


CompetencyState IsRevoked BOOL The learner has been punitively removed from being considered 
competent (i.e. revocation of credential) 


CompetencyState EffectiveDate ISO8601 When was the competency asserted or credential conferred (see 
Figure 14 for states) 


CompetencyState PendingDate ISO8601 For Credentials, when does a currency requirement need to be 
evaluated 
ConfigurationRecord SequencelD Long Record entry ID for changes 
ConfigurationRecord TimeOfChange ISO 8601 Time at which data was updated 
ConfigurationRecord AuthorityForChange URI (Identity Group or Person Who or what agency authorized the change 
Handle) 
ConfigurationRecord RecordType String Class name or record ID where change occurred 
ConfigurationRecord AttributeChange NVPS Array List of attribute/field, datatypes and values of change 
Device DeviceName String A common name for referring to the device on the screen 
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Class Attributes Data Type Definition 

Device IPAddress IPv4 array IP or IP range of device connected to the federation 

Device Type String Common name for type of device, e.g. mobile device, desktop client, 
simulation I|OS, Player Unit, Exercise Control, etc. 

Device Interface URL Endpoint universal Resource Locator for REST communication 

Device MACAddress Formatted String As required for cybersecurity (i.e. sticky-MAC), the physical hardware 
address of the device 

Device PhysicalLocation String May be updated in future iterations with GPS information, otherwise 
a way of locating the device, building, room, work site, etc. 

Device Certificate FIPS140.2 As applicable for cybersecurity, the digital certificate authorizing the 
device for operation on the network 

LearnerTask ScheduledEvent URI (Learning Experience Reference for the activity/content/competency tuple that the task 

Handle) represents 

LearnerTask ScheduledTime ISO8601 DTG of when the task was assigned 

LearnerTask SuspenseTime ISO8601 DTG of when the task must be completed by 

LearnerTask Authorized URI(Person:handle) Reference of the person who authorized the training even to occur 

LearnerTask Assigner URI(Person:handle) Reference of the person who assigned the learner to the event (may 
be the learner if it was requested) 

NVPS UserDefinedAttribute String Name of the locally defined data element 

NVPS UserDefineDataType DataTypeEnum List of possible data types 

NVPS Value Data Type Array Value placed in the field constrained by data type 

NVPS Scope String The level or command designation at which the specific attribute list 
is managed through governance (e.g. agency, ODNI, USAF, JICPAC) 

OCSTD ReferenceDocument URI(Content) The volume or reference ID of the occupational standard reference ID 

OCSTD InDocumentReference {String Internal section or subsection number 

Taxon Framework String The name of taxonomy or learning model (e.g. Bloom, Merrill) 

Taxon Enumerated Level Int Array Preserves a hierarchical structure or tensor (array index>1) for the 
taxonomy 

Taxon Enumerated List String Array The concepts filling a given level or cell for the taxonomic model 
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Attributes 


Data Type 


Definition 


xAPI Actor URI (Person: object Handle) The person or system that had the experience; refers to object from 
user and group management 

xAPI Verb URI (Profile: object handle) The action taken by the actor; refers to concept in Profile 

xAPI Object URI (Profile: object handle) The activity in which the action was taken; refers to concept in profile 

xAPI Timestamp ISO8601 When the action was taken 

xAPI Stored ISO8601 When the LRS stored the xAPI activity statement 


Language 


RFC5646 


The language code for the activity 


xAPI Result ResultEnum The grade or success of the activity 
xAPI Context ContextEnum The caveats explaining the activity (as defined in the index to support 
the competency object in the educational alignment) 
xAPI Authority URI (Person object handle, The person or system which authorized the creation of the 
interest group object handle or | experience record 
activity metadata object handle) 
xAPI Attachments AttachmentsEnum Any allowable files which are included with the activity 
xAPI Version String The xAPI version used (automatically populated by LRS) 
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xAPI Statement 


version: ### Key 
stored: mask_ISO8601 
timestamp: masks ISO8601 
ID: IRI 


mask_ISO8601 = timestamp 
IRI = Internalized Resource ID 
enum = enumerated data 

$ = String 

BOOL= boolean 


objectType: enum 


Eleeel bal 


homepage: URL 
name: $ 


name: $ 
objectType: enum 


account 


homepage: URL 
name: $ 


id: URI 
objectType: enum 


display 


definition 


object 


id: URI 
objectType: enum 


display 


definition 


success; BOOL 

com pletion: BOOL 
response: $ 

duration: mask_ISO8601 


scaled: -1-1 
min: < max 
max: > min 
raw: min — max 


registration: UUID 

instructor: Agent 

team: Group 

revision: $ 

platform: $ 

language: enum 

stat ement: StatementReference 


id: URI 
objectType: enum 


Figure 5. DIV-3 Physical Data Schema. 
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3.0 OPERATIONAL ViEWws (OV) 
3.1 OV-1 Operational Overview 


The graphic shown in Figure 6, depicts the relationship between the building blocks of the future 
learning ecosystem, its enclaves and federations, and the stakeholders that provide value to or receive 
value from deploying, integrating or using the data and computational resources within those enclaves 
and federations. The figure is organized to start with Initial Operational Capability (IOC) capabilities in 
the lower left hand and progressively field or migrate data and services to achieve the Final Operational 
Capability (FOC) in the upper right corner. 


3.2 OV-2 Operational Resource Flow Description 


The graphic in Figure 7 shows the various stakeholder types and the data (system digital 
communications) and information (process generated organizational data, managed through 
governance structures) exchanged between them. The elements on the right are the initial targets for 
migration, and as the capabilities mature, more and more stakeholders will be defined and integrated 
with processes and digital interfaces. The interface icons show the governance relationships, which are 
detailed in the DIV-1, Section 2.1 


3.3 OV-4 Organizational Chart 


The graphic in Figure 8 shows partial agency and department organizations that are typical of the 
agencies who have data equities for the TLA roles that were defined in the main report. In practice, each 
agency will determine its own processes and it’s likely that these will change over time in response to 
the TLA and other business realignment initiatives. 
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TLA - Total Learning Architecture - The Future Learning Ecosystem 


The culmination of several years research: 
The Total Learning Architecture 
- integrates new technology: 
* cloud computing; 
* machine learning; 
* data analytics; 
- streamlines management systems, 
- leverages advances in learning science, 
- removes redundancies, latencies, and 
inefficiencies in the legacy systems, and 
- integrates disparate data, services, vendors, 


and processes used to train and educate ~ 

the force. —h 
- This enhances the lethality of the — 

war-fighter through improved training and 

education that is traced to and validated by Content Managers 


unit readiness and mission & Competency Managers 
accomplishment. 


learning resources: 
activities and 
content 


Each Learning Command’s 


Bp Wero)enle)ir-lal meen) eel ie) ar-|| 
assets, data, and services 
create a learning enclave 


ade) (ent) ol er] (er- 1t(0) a1 
ME) -laTe[-l x0 ME lao Melo) {-1ear-laler- 
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Figure 6. OV-1 High Level Operational Concept. 
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Figure 7. OV-2 Operational Resource Flow. 
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Figure 8. OV-4 Organizational Relationships. 
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3.4 OV-6a Operational Rules Model: Business Rules 
3.4.1 Cybersecurity 


e The TLA shall store local data related to a user with anonymized identity references. 

e Each organization shall assign a group of individuals who are authorized to evaluate and register 
content and activities. 

e TLA implementations shall leverage industry best practices for back-end services for authentication, 
virtualization and storage. 

e Each learning organization shall assign a trusted group of individuals who can approves requests for 
event scheduling or validate observations and self-reports. 

e The LRS shall be network discoverable and auditable (enforce non-repudiation of serial entries and 
identity resolution). 

e =Any organization or group of organizations shall implement a federation using the TLA federation 
negotiation process, including cybersecurity policies to establish ATCs. 

e Systems shall use a globally unique UUID marker to associate PII with local records. 


3.4.2 Competencies 


e Each DoD Service Secretary or designate shall assign ownership of jobs and performance 
requirements for each job. 

e Jobs shall have assigned Competency Frameworks. 

¢ Competency Frameworks may include sub-frameworks shared from other competency owners. 

e Competency Frameworks shall be network discoverable and shareable. 

e The Competency System shall be able to validate frameworks against human/systems integration- 
system design, operational, and readiness data. 

e =6Any organization implementing TLA standardized activity, content and competency management 
services shall comply with the TLA interface specification and Functional Requirements Document. 


3.4.3 Data Governance 


e Every data element related to education and training shall have a single authoritative source. 

e Data interoperability shall be ensured using governance procedures to maintain semantic and 
protocol compatibility. 

e Each service shall assign one or more governance bodies to develop schema for Name-Value-Pair 
Sets describing user settable metadata to ensure semantic interoperability. Governance bodies will 
be severable by community of practice as defined. 

e Each category of metadata shall be assigned the proper scope and associated governance owner. 

e The Enterprise Course Catalog shall be parsed from applicable content metadata across all agency 
activity indices. 

e ©The activity indices shall be network discoverable and searchable. 

e Each organization’s Learner Profiles shall be discoverable. 

e User identity tokens shall be universally discoverable but protected. 

e LP credential updates can be forwarded to a centralized HR system. 

e Any organization implementing TLA standardized core data (Experience Index, Competency 
Frameworks, Learning Record Store, Learner Profile) shall comply with the industry-standard 
metamodels defined In the ADL Initiative Policy Framework (DODI 1322.26). 

e =©Any organization implementing TLA standardized core data shall comply with all appropriately 
scoped data governance for the establishment of metadata sets and data labeling. 
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Figure 9. OV6b Operational Rules Model: State Transition. 
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4.0 SYSTEM AND SERVICE ViEWs (SV/SvcV) 
4.1 SV-1 Notional System View of a Single Enclave and its Enterprise Asset Connections 


Figure 10 shows a notional deployment of actual databases and computational assets in an enclave, 
connected or federated learning devices with their TLA boundary and edge communications and 
services, as well as connections to enterprise assets. 


4.2 SvcV-1 Service Level View of Enclaves and Federations Within the Ecosystem 


Figure 11 shows a service-based view of the relationship between inter and intra enclave assets within a 
federation and across the ecosystem. 


4.3 SvcV-4 Service Functions and Interfaces Within an Enclave 


Figures 12, 13, 14, 15 and 16 include diagrams that show the high-level functions allocated to each 
microservice group, along with function block diagrams that provide details. These functions are 
captured in the Functional Requirements Document. 


4.4 SvcV-5 Operational Activity to Services Mapping 


Table 7 shows the way deployment of specific services provide value to or require input from the 
various stakeholders by category. This document can be used to define the deployment, and migration 
plan as well as stakeholder engagement plans. 
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Figure 10. SV-1 System Context Diagram. Shows internal components of notional enclave and connections to enterprise assets and notional connections across the ecosystem. 
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Figure 11. SvcV-1 Service Context Diagram. 
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Figure 12. SvcV-4 Services Functionality (Overall Logic Arrangement). 
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Figure 13. SvcV-4 Services Functionality (Event Management). 
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Figure 14. SvcV-4 Services Functionality (Activity and Resource Registry). 
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Figure 15. SvcV-4 Services Functionality (Competency Management). 
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Figure 16. SvcV-4 Services Functionality (Identity Management). 
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Table 7. SvcV-5 Operational Activity to Services. 
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Manage Decision Support Analytics X X X O X X O O X X X X 
Maintain Policy Framework O X X 
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Maintain Data Federation X X X X X X X X X O X X X 
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Synchronizing Enterprise Course Catalog X X X X X X X 
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Evaluate Learners X X X X X X X X X X X X X X X 
Register New Activities O O X X X X X X X X 
Manage Credentials X X X X X X X X X X X X X X X 
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4.5 SvcV-10a Services Business Rules 
4.5.1 Minimum (TLA Level 1 & 2) Compliance 


e Learning Commands must maintain records in a transactional Learning Record Store (LRS). 

e Learning Record Providers (LRP) are edge systems that represent a learning activity which provides 
the ability for a learner to experience one or more pieces of instructional content 

e The instructional content shall be aligned to address or assess an element of an educational or 
Competency Framework. 

e = This alignment may be monotonic, hierarchical, or manifold in nature. 

e TLAcompliant systems shall utilize the xAPI specification for recording learning experiences and 
paradata about learning experiences. 

e LRP Systems may have their own unique xAPI profile and internal architecture. 

e LRP systems must register a profile, which may be composed of existing profiles. 

e LRP systems are responsible for any branching, sequencing, and micro adaptation within the 
content. 

e LRP systems shall manage their own human performance assessment. 

e LRP may archive xAPI statements according to any profile in their own “noisy” LRSs. 

e The system owner or controlling agency for each enclave shall be responsible for obtaining the ATO 
for the enclave. 

e The LRP shall be able to normalize human performance data to the states and verbs defined in the 
TLA MOM profile for LRP that support self-directed, self-determined, or self- regulated learning. 


4.5.2 TLALevel 3+ Compliance 


e TLAcore services shall be able to interface with operational data and human resource system data 
using an industry standard or open source data or interface specification. 

e TLA core data shall be federated using prescribed TLA specifications and standards. 

e TLAimplementation data topology shall be maintained to optimize retrieval time, security, and data 
integrity. 

e _LRPs shall provide boundaries that normalize xAPI messages sent to the Transactional LRS to the TLA 
MOM verb states. 

e The TLA shall include components that satisfy core services, core data and expose required 
interfaces within a TLA enclave. 

e The TLA core services shall include activity management, content and resource management, and 
competency management. 

e The TLA core data shall include the Learner Profile, the Transactional LRS, the Competency 
Framework, the Experience Index. 

e The core services shall allow federation of data between the noisy/transactional/authoritative LRS, 
Experience Index,” Learner Profile and Competency Framework to external data using the REST API. 

e TLAcore services shall comply with the TLA MOM profile for data communication between services. 

e TLAcore services shall manage federation state only through the TLA MOM. 

e TLAcore services shall manage federation state independent of any edge systems. 

e TLAcore services shall expose a RESTful API as per the TLA interface specification. 

e The Experience Index shall comply with the content and resource metamodel. 

e The Competency Framework shall comply with the competency metamodel (IEEE P1484.20.1) 

e The Learner Profile shall comply with the learning profile metamodel. 

e _LRPs are to be edge systems in the TLA topology. 
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e LRP edge systems shall normalize communication to core services conforming to the TLA MOM IEEE 
P9274.3.1. 

e LRP Systems shall be responsible for their own adjudication of student behavior against the 
standards defined in the Competency Framework. 

e LRP edge systems shall report applicable context values as defined in the Competency Framework. 

e TLAcore services shall expose interfaces to allow for agility in presentation technology. 

e =6Alluser interfaces shall be edge systems. 

e The core service data shall include severable data structures to support the Transactional LRS. The 
Learner Profile, the Experience Index, and the Competency Framework shall report all their data to 
the Transactional LRS. 

e The API shall support secure socket layer and encryption. 

e The core services should use a publication subscribe messaging topology internally. 

e The core services shall use a secure, non repudiable identity hashing system to store user identity in 
records. 

e The core services shall support reconciliation of noncontiguous identity hash systems between 
federates. 

e The core services shall support use of a globally unique and non repudiable identity management 
system in the overall learning ecosystem. 

e The core services shall support server and network virtualization services maintained external to the 
enclave. 

e The core services shall support tagging and encryption of all data in transit for systems that require 
that level of robustness. 

e The core service data shall support the minimum metadata and data labeling requirements for 
global interoperability. 

e The core services may assign locally defined and maintained metadata and data labeling 
requirements. 

e Locally assigned metadata and data labeling standards and enumerations shall be documented in an 
enclave API, made available as part of the cybersecurity MOA process. 

e The enclave API shall comply with the TLA DIV-2 logical data model. 

e The core services must not include an individual’s PP! when producing their xAPI statements. 


4.6 SvcV-10b Services State Transition Diagram 


Figures 17, 18, and 19 include diagrams that describe the state of the learner acting within a TLA 
compliant enclave, as the collection of microservices proposed in the TLA systems concept represent a 
stateless federation (although the actual components used to create the service may have state and 
may be different between installations, pending service/agency unique solutions). Essentially this 
describes the detailed life cycle for the TLA MOM. 


4.7 SvcV-10c Services Event Trace Diagram 


Figure 20 shows the nested sequence of data filtering and analysis, learning planning, and learning 
execution control within the five control loops that were described in the report. 
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Figure 17. SvcV-10b Services State Transition Diagram (Learner Career States). 
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Figure 18. SvcV-10b Services State Transition Diagram (Learning Event States). 
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Figure 19. SvcV-10b Services State Transition Diagram (Learning Record Provider States). 
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Figure 20. SvcV-10c Services Event Trace Diagram. 
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5.0 
5.1 


STANDARDS Views (STDV) 


StdV-1 Standards Profile 


5.1.1 TLA Specifications and Standards 


Research is continually performed to evaluate how standards and specifications might be used across 
the future learning ecosystem, at various stages of a career trajectory, and across institutional 


boundaries. Table 8 provides a mapping of relevant specifications and standards for each TLA 
component or service. This table summarizes the candidate standards and specifications utilized or 


investigated in 2018 and 2019. It represents the current state of specifications under evaluation. The 
StdV-2 view provides an objective end-state of specifications that were down-selected from this list for 
the 2019 Reference Implementation, as well as the proposed standards that will complete the TLA policy 


framework. 


Table 8. Summary of TLA Specifications. A breakdown of each candidate specification that was evaluated in 2018 
to show how it was used. The specifications are grouped and listed according to the TLA component it is aligned 
with. There is an expectation that TLA components and their outputs adhere to the referenced specifications. 
Supporting vocabularies, rules, and software ensure this process. 


TLA Component 


Standard/Specification 


Transport 


Data Store / Registry 


Activity Registry LRMI JSON Experience Index 
IEEE 1484.12.1 LOM XML Experience Index 
Schema.org Vocabularies Experience Index 
Activity Stream IEEE xAPI HTTP / HTTPS with JSON Learner Record Store 
payloads 
SISO HPML Human Performance 
Data Model 
Competency ASN™ CaSS 
Management CASE ™ CaSS 
O*Net CaSS 
Medbiquitous CaSS 
RCD CaSS 
Credentials CTDL Credential Authoring Credential Registry 
Language 
IMS Global Open Badge 
Data Analytics and Multiple DAVE Algorithms, DAVE Dashboards, TLA Portal 
Visualization Visualizations, Data Cards 
Environment 
Learner Profile CaSS Proprietary 


Enterprise Learner Record 


OICD, JSON, OpenBadge2.x 


Learner Profile 


Comprehensive Learner Record 


Proprietary 


Learner Profile 


Identify Management / 
Single Sign On 


OpenID Connect (profile of 
OAuth2.0) 


HTTP / HTTPS with JSON 
payloads 


TLA Common 
Education and Training 
Portal 


Learner Record 
Provider - eBooks 


ePUB 3, xAPI 


The following sections describe the candidate specifications and standards for each TLA component or 


service, providing an evaluation of capabilities and recommended extensions to support TLA 
requirements with additional insights on how they complement or compete with other specifications. 
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5.1.2 Activity and Resource Registry 


The Activity and Resource Registry includes an Experience Index that stores metadata about each TLA 
learning activity. The approach used in the 2018 Reference Implementation relied heavily on the 
Learning Resource Metadata Initiative (LRMI) specification. While LRMI shows promise for tagging new 
content, there are thousands of learning resources tagged using metadata schemas managed under 
different organizations. Given the current state of technology, the TLA needs to be compatible with 
many of the more common metadata formats. 


All data formats currently being investigated only support human-readable descriptions and don’t 
consider the type of metadata generated by modern machine learning algorithms. These systems 
generate metadata based on a statistical analysis to describe concepts such as engagement, 
effectiveness, or any number of other descriptors that would be useful to other TLA components. In the 
future, an Activity Registry will include data and connected services that discover, catalog, and store 
information about TLA compatible learning activities. A primary goal of the 2019 research was to 
understand how the Activity Registry can be used to drive an Enterprise Course Catalog, populated by 
different education and training organizations. Other 2019 research included the identification of 
approaches for integrating paradata into the Activity Registry. The following metadata standards are 
being looked at to describe TLA content. 


5.1.3. Learning Resource Metadata Initiative (LRMI) 


The LRMI¢ is a common metadata framework developed by Creative Commons and the Association of 
Educational Publishers (AEP) for describing and tagging of educational resources in web-based 
instruction and training. The LRMI metadata schema was adopted by Schema.org in April 2013. It allows 
anyone who publishes or curates educational content to use LRMI markup to provide rich, education- 
specific metadata about their resources with the confidence that this metadata will be recognized by 
major search engines. 


The LRMI specification is a collection of classes and properties for metadata markup and description of 
educational resources. The specification provides clear guidance of the terms within and how to use 
them both in coding and through descriptive language to be followed by those implementing it. The 
attributes defined by the metadata are clear and the rules surrounding the specification drive 
interoperability. The LRMI 1.1 specification is stable and has seen significant adoption. 


The LRMI specification includes an “AlignmentObject” as part of the specification. This object describes 
an alignment between a learning resource and a node in an educational framework (e.g., Competency 
Framework). This feature was used extensively in the 2018 Reference Implementation. The 2018 
Alignments included educational audience, educational use, competency object, simple or complex, 
interactivity level, and others. While limited, these use cases proved the ability of the AlignmentObject 
to serve its intended purpose. 2019 research has focused on building out a minimum set of metadata 
requirements that should be implemented for any future learning activities or content. 


§ http://Irmi.dublincore.org/ 
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Figure 21. Learning Object Metadata. LOM comprises a hierarchy of elements that includes nine categories, each of which contains sub-elements. The semantics of an element are determined by its context: they are affected by the parent or container element in the 
hierarchy and by other elements in the same container. 
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5.1.4 lEEE 1484.12.1 Learning Object Model (LOM) 


As shown in Figure 21, LOM’ provides a broad framework for describing learning objects to facilitate 
search, evaluation, acquisition, and use. LOM is used by the SCORM reference model to provide 
descriptive information about a learning object, including the title, author, description, keywords, 
educational objective, and other relevant information. The LOM data model specifies which aspects of a 
learning object should be described and what vocabularies may be used for the descriptions; it also 
defines how this data model can be amended by additions or constraints. The metadata is hierarchical, 
with a root and leaf nodes. Alignment can be connected to a discipline, idea, prerequisite, educational 
objective, accessibility, restrictions, educational level, skill level, security level, or competency. 


The purpose of this standard is to allow the creation of LOM instances in XML. This allows for 
interoperability and the exchange of LOM XML instances between various systems. When implementing 
the LOM as a data or service provider, it is not necessary to support all the elements in the data model, 
nor need the LOM data model limit the information which may be provided. 


The creation of an “application profile” allows a community of users to specify which elements and 
vocabularies they will use. Elements from the LOM may be dropped and elements from other metadata 
schemas may be brought in; likewise, the vocabularies in the LOM may be supplemented with values 
appropriate to that community. For example, the Healthcare LOM®, developed by the Medbiquitous 
consortium extends the LOM standard and provides custom vocabularies for some metadata elements. 


The attributes defined by the metadata are clear and the rules surrounding the specification drive 
interoperability. The LOM specification (1484.12.1-2002) is stable and has seen significant adoption. ADL 
Initiative stakeholders have thousands of SCORM courses that have been encoded with LOM metadata 
so any TLA component that relies on metadata needs to be compatible with LOM. As with most 
metadata formats, consistency of quality metadata is a known issue. 


5.1.5 Dublin Core Metadata Initiative (DCMI) Metadata 


The Dublin Core Schema’ is a set of vocabulary terms that describe digital resources and physical ones 
such as books or CDs, and objects like artworks. The Dublin Core Metadata Element Set is a vocabulary 
of 15 properties used for resource description. It is part of a larger set of vocabularies and specifications 
maintained by the DCMI. DCMI metadata may be used for multiple purposes, including simple resource 
description, combining metadata vocabularies of different metadata standards, providing 
interoperability for metadata vocabularies in the linked data cloud, and Semantic Web implementations. 


The full set of vocabularies also includes sets of resource classes, vocabulary encoding schemes, and 
syntax encoding schemes. The terms in DCMI vocabularies are intended to be used in combination with 
terms from other, compatible vocabularies in the context of “application profiles” and the DCMI 
Abstract Model. As part of an extended set of DCMI Metadata Terms, Dublin Core became one of most 
popular vocabularies for use with the widely used Resource Description Framework (RDF), more recently 
in the context of the Linked Data movement. 


7 https://ieeexplore.ieee.org/document/1032843/ 
8 https://www.medbiq.org/working_groups/learning_objects/HealthcareLOMSpecification.pdf 
° http://www. dublincore.org/specifications/ 
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The DCMI assumed stewardship of the LRMI specification in 2014. The Dublin Core schema is relevant 
because it is widely used. While the educational alignment aspects of DCMI are not as robust as LOM or 
LRMI, the ability to tie into additional vocabularies provides an attractive mechanism to identify and 
catalog content from different communities of interest, industries, or demographics. 


5.1.6 Schema.org Vocabularies 


Founded by Google, Microsoft, Yahoo and Yandex, Schema.org vocabularies?” are developed by an open 
community with a mission to create, maintain, and promote schemas for structured Internet data. Each 
schema.org Item Type has its own set of descriptive properties. The broadest Item Type is “Thing,” 
which has four properties (name, description, URL, image). More specific “Types” share properties with 
broader “Types.” For example, a “Place,” “Person,” or “CreativeWork” is a more specific type of “Thing.” 


LRMI’s adoption into schema.org vocabularies provide many benefits. In theory, nearly any schema.org 
“Thing” could be a learning resource. Therefore, LRMI addresses those metadata properties that 
distinguish content when it is deliberately used for learning. This was done by adding learning-resource 
properties to key root types (e.g., CreativeWork). Currently, CreativeWork properties include 
“Educational Use” and “Educational Alignment”. A more specific type of CreativeWork includes a 
“Course”™ which may be offered as distinct instances at which take place at different times or take place 
at different locations or be offered through different media or modes of study. An educational course is 
a sequence of one or more educational events and/or other types of CreativeWork which aims to build 
knowledge, competence or ability of learners. 
alignment 


Type educational 
Framework 


target URL 


target description 


Creative Work Alignment Object 


_educational. 
Alignment e 


Educational 
Framework 


Figure 22. Alignment between a Schema.org Creative Work and a node in an Educational Framework’”, The 2018 
TLA Reference Implementation used LRMI’s Alignment Object to reference a Competency Framework that provided 
a structured description of required knowledge, skills, abilities, and their interrelated relationships. 


As shown in Figure 22, a learning activity is a schema.org > CreativeWork > “Thing” that inherits the 
properties that every schema.org Thing has and can be used to support multiple forms of alignment that 
are possible between a resource and an educational framework. The AlignmentObject can also be used 
to distinguish between resources that teach and assess. This presents the ability to collect paradata 
about how different types of content apply to different instructional domains and enables new insights 
into which content is more effective. 


10 https://schema.org/ 
1 https://schema.org/Course 
? https://blogs.pjjk.net/phil/explaining-the-Irmi-alignment-object/ 
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5.1.7. Activity Streams 


The FLE offers a wide range of learning content across the continuum of learning. An Activity Stream is a 
list of events generated by individuals, groups, applications, or learning activities that provide details 
about the ongoing experiences to other TLA components. 


The types and variety of activities that are used for learning can often be associated with a specific 
delivery modality. Instructor-led classroom training will create one set of instructional activities, while 
serious games and simulations have the potential of generating a completely different set of activities. 
This has potential to have two similarly named activities with two different contexts for how those 
activities are being applied and the experiences they encompass. A common vocabulary is necessary to 
ensure all learning activities accurately describe the experience. By formalizing this vocabulary, a set of 
attributes and rules about the data is established, such as how they are stored, retrieved, and accessed 
by other components, systems or activities. 


The different activity stream specifications investigated for inclusion in the TLA are similarly structured. 
Each specification includes serialized data streams that consist of statements about activities. Such 
statements typically involve a subject (the person doing the activity), a verb (what the person is doing), 
and a direct object (what the activity is being done to or with). 


The subject of an activity is nearly always the learner but could foreseeably be an instructor, cohort, or 
other. The direct object of an activity is presented differently depending on its context. Verbs must 
conform to a common vocabulary; otherwise different organizations will use different verbs to describe 
the same activity or the same verb to describe different activities. 


5.1.8 Experience API (xAPI) 


The xAPI?? specification is in the process of becoming a standard through the Institute of Electrical and 
Electronics Engineers — Learning Technology Standards Committee (IEEE-LTSC)**. The xAPI specifies a 
structure to describe learning experiences and defines how these descriptions can be exchanged 
electronically. The main components of xAPI are the data structure called Statements and the data 
storage/retrieval capability called the Learning Record Store (LRS). The xAPI specification has stringent 
requirements on the structure of this data and the capabilities of the LRS. Statements are data triples 
that use an Actor, a Verb, and an Object to describe any experience. Each statement also includes 
timestamps and unique, resolvable identifiers. The transport is HTTP/HTTPS with JSON payloads. 


Any enabled device can send xAPI statements including mobile phones, serious games and simulations, 
CPR dummies, and any number of other learning systems. The xAPI Profile Specification’ offers a 
common way to express controlled vocabularies across these different mediums, provides instructions 
on the formation of xAPI statements, and describes patterns of xAPI usage that provides additional 
context to a domain, device, or system. The xAPI Profiles Specification also adds tools to support 
authoring, management, discovery and/or adoption, including additional data elements and properties. 


13 https://github.com/adInet/xAPI-Spec 
44 https://www.tagxapi.org/ 
4 https://github.com/adInet/xapi-profiles 
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An LRS is the implementation of the server-side requirements associated with the xAPI specification. It is 
the application interface for storing, accessing, and often visualizing the data about learning 
experiences, activities, and performance. The LRS is essentially a Web service that leverages a REST API 
for the storage and retrieval of xAPI data. A learner’s xAPI data and transcripts stored within one LRS can 
be extracted and sent to other LRSs, enabling learning experiences to follow them from one organization 
to another. 


An LRS could be optionally integrated with any application such as an LMS, Human Resources system, or 
could serve as centralized data store in an enterprise learning ecosystem. Third party applications which 
send or retrieve learning activity data will interact with the LRS as the data store for xAPI data. From this 
perspective, an LRS is also required to validate the format of the statements as many of the 
requirements in the xAPI specification are targeted toward the LRS component. 


5.1.9 Caliper 


Developed by the IMS Global Learning Consortium, Caliper is essentially a competitor to xAPI, in that it 
records learning event data. However, Caliper is more normalized than xAPI and while that typically 
means a faster or more compact data transmittal and storage format, it represents a more brittle 
architecture. Caliper was removed from review in 2019. 


5.1.10 W3C Activity Streams 2.0 


The W3C Activity Streams” is an open format specification for activity stream protocols, which is similar 
in function to the xAPI in the TLA topology. In the 2019 Reference Implementation, “activity streams” 
are tracked using the MOM profile of xAPI. The implementation of linked data does not use JSON-LD but 
is more thread-safe than JSON-LD in the open ended FLE. 


5.1.11 Human Performance Markup Language (HPML) 


Developed by the Simulation Interoperability Standards Organization (SISO), HPML?” is an XML-Schema- 
based language intended to support the evaluation of individuals and teams as they perform their job 
functions. HPML provides schemas for organizing the information relevant to defining performance 
measurements, including computations, measures, assessments, results, and instances/periods. 
Specifically, it is an XML based language designed to express performance measurement concepts ina 
format that is both machine and human readable. 


HPML enables the explicit combination and transformation of performance data into performance 
measurements and assessments. These are decoupled from a specific training system but can be used to 
specify the data elements to be collected from a system, the calculations to use to process the data, and 
when to produce performance measurement results. As shown in Figure 23, HPML provides a flexible 
framework for configuring and executing performance measurement and assessment. The schema is 
separated into six distinct groups that make up the core components of HPML and can be expanded 
with additional links in the schemas. 


16 https://www.w3.org/TR/activitystreams-core/#documents 
17 https://discussions.sisostds.org/index.htm?AO=SAC-PDG-HPML 
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Figure 23. HPML High-Level Conceptual Architecture. HPML includes a Human Performance Data Model with rules 
to dictate how performance data are measured, computed, and assessed. It is agnostic on the kinds of data used. 


HPML as a specification includes both the markup language used to author the activity stream and the 
“Human Performance Data Model” used to measure and assess performance based on the incoming 
stream of data. In the 2019 Reference Implementation, all adjudication of performance has been moved 
to edge systems and away from core data. Thus, HPML may still be a part of an edge system solution, 
but it is not part of the TLA policy framework per se. 


5.1.12 Competency Management 


Competency management requires the generation of rich and traceable data about learning 
experiences, how they relate to skill proficiency, and the knowledge, skills, abilities, and behaviors that 
individuals need to do their job. Competencies describe specifically what people need to do to be 
effective in their roles, and it clearly establishes how their roles relate to organizational goals and 
success. Each individual role has its own set of competencies needed to perform the job effectively. 


Competency Based Learning represents a transition from curricula focused on abstract knowledge and 
pursuit of certificates, to curricula built around authoritative performance indicators that guide learning 
experiences based on challenges in the workplace that unlock human potential. Proficiency is a complex 
and critical concept that requires relevant, trusted data that can be used as evidence about an 
individual’s mastery against a specific set of competencies. A Competency Framework is a structure for 
defining the knowledge, skills, abilities, and attitudes (or other characteristics) required to do a job. 
Competency Frameworks are widely used in business for defining and assessing competencies within 
organizations in both hard and soft skills that jointly define successful job performance. There are 
numerous Competency Frameworks available and numerous specifications that drive them. 


The 2019 effort focused on determining a meta-metamodel for describing competencies as a series of 
Directed Acyclic Graphs (DAG) and thus the Reusable Competency Definition (RCD) objects provide a 
mathematical formalism for describing competencies, rather than a framework of competencies 
themselves. The subordinate frameworks, ASN, CASE, O*net, MedBiquitous, etc. can be represented 
within this formalism, as appropriate for the communities of practice they represent. 
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5.1.13 Achievement Standards Network (ASN) 


ASN provides access to machine-readable representations of learning objectives and curriculum 
standards”. It provides an RDF-based framework based on the syntax-independent DCMI abstract 
model (DCAM). The DCAM is intended to support development of Dublin Core Application Profiles 
(DCAP) of which the ASN DCAP is an example. The ASN framework is made up of “Standards 
Documents,” which represent the overall Competency Framework; and “Statements,” that represent 
the individual achievements within the overall framework. A set of core properties define the 
relationships between the two in terms of an “Entity Relationship” model. Structural relationships 
replicate the relationships between different components of the Standards Document and semantic 
relationships define the relationships of meaning between statements (e.g., assertion of equivalence). 


ASN is designed for interoperability and open access to learning objectives. It has seen wide adoption in 
the K-12 community. The ASN Description Framework (ASN-DF) also provides the means to create "ASN 
profiles" through inclusion of additional properties and classes from other namespaces and definition of 
constraints on data patterns. ASN-DF provides a small vocabulary of classes and properties and a set of 
mechanisms for tailoring an ASN Profile based on the Dublin Core's conceptualization of application 
profiles and description set templating as well as refinements to these conceptualizations developed by 
the U.S. Library of Congress to support Bibilographic Framework (BIBFRAME) profiling. 


5.1.14 Competencies and Academic Standards Exchange (CASE) 


The IMS Global Learning Consortium created the CASE specification’® to define how systems exchange 
and manage information about learning standards and competencies in a consistent and digital way. 
CASE connects standards and competencies to performance criteria and provides a way to transmit 
rubrics between various platforms. Within CASE, the underlying structure of both a competency and an 
academic standard are represented using the same data model. The data model is composed of three 
core constructs: 


e The root definition document — the “CFDocument” is the top-level structure that holds the set of 
statements that define the individual competencies/academic standards. 

e The set of composition statements — the “CFltem” is the set of statements into which the top-level 
competency/academic standards have been decomposed. 

e The rubric — the “CFRubric” defines how mastery of the competency/standard can be achieved. This 
requires the definition of specific criteria used for each of the scores that can be awarded during an 
assessment. 


The CASE specification defines internal relationships between statements like “Parent/Child,” 
“Precedes,” “Is Related To,” “Exemplar,” and “Is Part Of.” All Competency Frameworks published in 
CASE format can be linked together in a network of equivalent or aligned competencies. Having 
universal identifiers for each competency makes it possible for any tools or applications to share 
information between systems. 


18 http://www.achievementstandards.org/ 
19 http://www.imsglobal.org/activity/case 
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5.1.15 O*Net 


The Occupational Information Network (O*NET)7° is sponsored by the U.S. Department of Labor, 
Employment and Training Administration (USDOL/ETA) to facilitate and maintain a skilled workforce. 
Central to the project is the O*NET database, which contains hundreds of standardized and occupation- 
specific descriptors on almost 1,000 occupations covering the entire U.S. economy. Every occupation 
requires a different mix of knowledge, skills, and abilities, and is performed using a variety of activities 
and tasks. As shown in Figure 24, the O*NET database identifies, defines, describes, and classifies 
occupations through Experience Requirements (Training, Licensing), Worker Requirements (Basic and 
Cross-Functional Skills), Occupation Requirements (Work Activities, Context), Worker Characteristics, 
Occupation Specific Information, and Occupational Characteristics. 


The value of the O*NET Database to the TLA is in leveraging the existing resources these government 
sponsored activities have already amassed. Their Military Transition search connects occupations in the 
O*NET database to other classifications systems within the Military. This is accomplished through data 
from the Military Occupational Classification (MOC) system at the Defense Manpower Data Center 
(DMDC). This capability is also available via web services. Other resources include “Careers in the 
Military” and links to Army, Navy, Marine Corps, and Air Force “Credentialing Opportunities On-Line” 
(COOL) projects. 
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Figure 24. O*NET Content Model. O*Net provides the framework that identifies and organizes the distinguishing 
characteristics of an occupation. The model defines an occupation as a standardized, measurable set of variables 
called "descriptors." This hierarchical model starts with six domains that expand to 277 descriptors. 


20 https://www.onetonline.org/ 
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5.1.16 MedBiquitous Competency Framework 


The MedBiquitous Competency Framework, ANSI /MEDBIQ CF.10.1-2012, is a technical standard for 
representing Competency Frameworks in XML, accredited by the American National Standards Institute. 
Organizations that publish Competency Frameworks can do so in this standard format, making it easier 
to integrate Competency Frameworks into educational technologies like curriculum management 
systems. The standard allows medical schools and other health profession schools to connect their 
curriculum, learning resources, and assessment data back to a common set of competencies, ultimately 
enabling competency-based views of the curriculum and of learner performance. The data model 
establishes relationships between “Competency Objects” that narrow or broaden the definition of 
overall competency. 


The MedBiquitous Competency Object specification provides a consistent format and data structure for 
defining a competency object, an abstract statement of learning or performance expectations, and 
information related to the statement. Statements can be learning outcomes, competencies, learning 
objectives, professional roles, topics, classifications/collections, etc. The Competency Object may 
include additional data to expand on or support the statement. This specification is meant to be used 
with the MedBiquitous Competency Framework specification. 


5.1.17 IEEE 1484.20.1 Reusable Competency Definitions (RCD) 


The current RCD standard”? defines a data model for describing, referencing, and sharing competency 
definitions, primarily in the context of online and distributed learning. The standard provides a way to 
represent formally the key characteristics of a competency, independently of its use in any specific 
context. It enables interoperability among learning systems that deal with competency information by 
providing a means for them to refer to common definitions with common meanings. 


This specification enables the storage of a Competency Framework in the form of a Dynamic Acyclic 
Graph (DAG) where competency objects define the knowledge, skills, conditions, and other factors that 
role up into an ability to do a job. The edges between each competency object inform the relationship 
between them and have the potential to identify whether they’re dependent, complimentary, 
conflicting, or other. This provides an extensible mathematical underpinning to a Competency 
Framework that accommodates the relationships defined in any other foreseeable Competency 
Framework. 


The Data Model for RCD”? was put together in September 2018 to revise the 2008 standard. The 
Competencies WG 20 intends to take the Credential Ecosystem Mapping Project’s mapping of 
competencies metadata and update RCD to represent the most common elements that are found in 
multiple standards addressing competencies and Competency Frameworks. The ADL Initiative is a 
participant in the LTSC working group and is keenly interested in the newer version of the RCD standard 
to describe the mathematical formalism of competency data from other sources. 


“1 https://www.medbig.org/working_groups/competencies/index.html 
22 https://ieeexplore.ieee.org/document/4445693 
3 http://sites.ieee.org/sagroups-1484-20-1/ 
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5.1.18 Credentials 


A credential is a testament of qualification and competence issued to an individual by an authoritative 
third party. Examples of credentials include academic diplomas, degrees, certifications, professional 
licenses, or other proof of occupational qualification. Within the TLA, credentials are an exchange 
format by a trusted party that formally encapsulate a set of “Competencies.” By their nature, they are 
linked to other TLA components such as Competency Frameworks, assessment rubrics, or Talent 
Management Systems. This requires the establishment of attributes and rules that allow TLA 
components to process and compare credentials in the same, interoperable way, particularly in Learner 
Profiles across instantiations of the TLA. This also requires a common way to describe, store, and access 
credentials in order to make fair comparisons of the achievement that was performed to acquire them. 


5.1.18.1 Credential Transparency Description Language (CTDL) 


CTDL” is a vocabulary comprised of terms that are useful in making assertions about a credential and its 
relationships to other entities. CTDL is modeled as a directed graph using RDF for describing data on the 
web. Like an activity stream, the "triple" is the basic grammatical construct in making CTDL assertions 
about "things" and is comprised of three simple components: a subject, a predicate and an object. This 
structure allows simple statements to enable a rich description of credential-related resources including 
credentialing organizations and specific subclasses of credentials such as Degrees, Certificates, and 
Digital Badges. The comprehensiveness of this specification makes it ideal for defining TLA credentials. 


5.1.18.2 Credential Registry 


The Credential Registry”° is both a repository of credential information and a set of services that make it 
easier to use that information. The Credential Registry works with CTDL to allow the storage of 
credentials that have been expressed using that specification. It also registers credentialing 
organizations that issue these credentials including universities, colleges, schools, professional 
associations, certification organizations, and more. 


Since it is strictly a registry, it only stores the description of a credential in the abstract sense and does 
not include any personal information or personally obtained credentials. This information will typically 
be stored in a Learner Profile. As a registry, it does allow users to see what various credentials represent 
in terms of competencies, transfer value, assessment rigor, third-party approval status and more. 


The Credential Engine project's developers are using DCAP to create systems that communicate virtually 
all aspects of credentials. The Technical Advisory Committee promotes collaboration across different 
standardization initiatives that are developing data models, vocabularies, and schemas for credentials 
and Competency Frameworks. The Credential Registry uses technology and CTDL to capture, link, 
update, and share information about credentials so it can be organized and centralized within the 
registry, made searchable by customized applications, and linked-to from anywhere on the open Web. 


4 http://credreg.net/ctdl/handbook 
5 https://www.credentialengine.org/credentialregistry 
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5.1.18.3 IMS Global Open Badges 


Open Badges” are visual representations of verifiable achievements earned by recipients. Open Badges 
is a technical specification and set of associated open source software designed to enable the creation 
of verifiable credentials across a broad spectrum of learning experiences. The Open Badges standard 
describes a method for packaging information about accomplishments, embedding it into portable 
image files as digital badges, and establishing resources for its validation and verification. Open Badges 
contain metadata about achievements such as who earned a badge, who issued it, and what it means. 
Open Badges 2.0 expands on this to include versioning, endorsements, and full use of JSON-LD. 


Open Badges are expressed as linked data so that badge resources can be connected and reference by 
other systems. This metadata can also be embedded within the graphic. The Open Badges vocabulary 
defines several data classes used to express achievements. There are three core data classes (Assertions, 
Badge Classes, and Profiles) that define mandatory and optional properties as well as the restrictions on 
the values those properties may take. Each Badge Object may have additional properties in the form of 
an Open Badges Extension, a structure that follows a standard format so that other users can 
understand the information added to badges. Assertions are representations of an awarded badge, used 
to share information about a badge belonging to one earner. 


5.1.18.4 T3 Innovation Network 


The T3 Innovation Network was launched in 2018 by the U.S. Chamber of Commerce Foundation and 
the Lumina Foundation to create a more responsive, dynamic, and equitable talent marketplace through 
the convergence of Web 3.0 technologies. Ten pilot projects were initiated in 2019 that covered topic 
areas like open data standards, comprehensive learner and worker records, shared competency 
infrastructure, and linked data to facilitate the technology infrastructure of the future. 


The ADL Initiative 2020 goals align with the T3 Network’s goals, specifically with three Pilot Projects: 
PP1- Data Standards Harmonization; PP3- Learner Record Standards; and PP5- Competency Data 
Exchange. Outcomes of these efforts include the development of robust learner, worker, and military 
use-cases. The projects support the mapping and harmonization of standards across the public-private 
talent marketplace, especially for the implementation and use of a comprehensive learner-worker- 
military, record. The conclusion of these pilot projects includes a plan for long-term sustainability, and 
governance/management for continued progress toward a more equitable talent marketplace. 


5.1.19 Learner Profile 


Learner Profiles exist in many systems and are extremely diversified in nature. It is envisioned that an 
FLE Learner Profile will house data about people across their entire career. Adult learners are 
characterized by distinctive personal attributes, knowledge, skills and competencies that they have 
gained in different contexts through a process of development and growth. A Learner Profile may also 
include demographic data, student interests, learning preferences, descriptions of preferred learning 
environments, inter/intra personal skills, existing competencies and those that need to be developed, 
socio-emotional health, and a myriad of other data. Learner Profiles are dynamic and will change over 
time. As student interests change and they become competent in new areas, the profile will update to 
reflect the latest “state” of the learner. 


26 https://www.imsglobal.org/activity/digital-credentials-and-badges 
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The 2019 Learner Profile is based on the attributes defined in the DIV2, but the Enterprise Learner 
Record (ELR) metamodel will extend to include the management of local and global values. Local values 
include persistent learner-state messages gathered from the MOM activity stream, local user attributes 
used for planning of local learning events, and local credential records, including the digitally signed 
Mom xAPI statements from the “chain of trust” and their association to competency “chains of 
evidence,” as well as the portable digital badge that is exchanged globally. A governance board for 
evaluation of local attributes to be elevated to global governance is a critical component to future 
learning ecosystem’s ELR/LP federation. 


5.1.19.1 Comprehensive Learner Record (CLR) 


IMS Global led the development of the CLR?’, formerly known as the Extended Transcript. It was 
originally designed to support traditional academic programs as well as co-curricular and competency- 
based education by capturing and communicating a learner's achievements in verifiable, digital form. 
The CLR contains data about different learning experiences and achievements including course 
descriptions, syllabi, rubrics, portfolios, performance evaluations, prior learning, internships, and other 
experiences or assessments. 


As a digital transcript, the CLR closely follows the registrar guidance of the American Association of 
Collegiate Registrars and Admissions Officers (AACRAO) to draw information from an institution’s 
Student Information Systems, LMS, or other internal databases.”® 


The CLR is due to be completed in 2019 but will likely evolve to help foster adoption across higher 
education institutions. Integration with existing institutional reporting systems and data structures will 
be critical in enabling this effort to succeed. The ADL Initiative will continue to monitor this effort to 
measure its applicability to the DoD and other government stakeholders. LIP will be evaluated for 
incorporation into the larger Enterprise Learner Record metamodel. 


5.1.19.2 Learner Information Package (LIP) 


The LIP specification”? provides a standard means for recording information about learners. It will allow 
information about learners, including their progress to-date and awards received, to be transferred 
between different software applications. LIP will be evaluated for incorporation into the larger 
Enterprise Learner Record metamodel. In this specification, Learner Information is separated into 11 
main categories, including: Identification, Qualifications/Certifications/Licenses, Accessibility, Activity, 
Goal, Competency, Interest, Transcript, Affiliation, Security Key, and Relationship. The LIP specification 
describes the data structures, XML binding, and best practices for formatting, storage, and exchange of 
learner information. 


The specification supports the exchange of learner information among LMSs, Human Resource Systems, 
Student Information Systems, and other systems used in the learning process. LIP is an older 
specification and its reliance on XML impacts the access speed for how these systems can load learner 
information. The FLE will require real time learner services that can send and update records on the fly. 


27 https://www.imsglobal.org/activity/comprehensive-learner-record 
28 https://library.educause.edu/~/media/files/library/2019/1/eli7164. pdf 
9 http://www.imsglobal.org/profiles/index.html 
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An Accessibility-for-LIP standard*? is extending the LIP Information Model to better define accessibility 
preferences for people with disabilities. This work is intended to benefit all learners in situations that 
require alternative modes of instruction, such as an extremely noisy environment where captions are 
needed for a video or language barriers. 


5.1.19.3 IEEE Public and Private Information for Learners (PAPI Learner) 


The PAPI Learner specification? is a multi-part standard that specifies the semantics and syntax for 
storing, retrieving, searching, and exchanging learner information. It defines elements for recording 
descriptive information about knowledge acquisition, skills, abilities, personal information, learner 
relationships, preferences, performance, and similar types of information. An important feature of the 
PAPI Learner standard is the logical division, separate security, and separate administration of several 
types of learner information. 


The PAPI Learner specification partitions learner records into six main information types that support 
extension: 


e Learner Contact Information — describes information related to administration. 


e Learner Relations Information — stores information about the learner’s relationships to other users 
of learning systems, such as teachers and other learners. 


e Learner Security Information — stores information about the learner’s security credentials, e.g. 
passwords, private keys, public keys, biometrics. 


e Learner Preference Information — describes preference information intended to improve human 
computer interactions and the automatic adaptation and personalization of systems to specific 
needs of the learner. 


e Learner Performance Information — is about the learner’s history, current work, or future 
objectives. PAPI performance information is primarily created and used by learning technology 
components to provide improved or optimized learning experiences. 


e Learner Portfolio Information — is a representative collection of the learner’s works or references to 
them that is intended for presentation and evidencing of his achievements and abilities. 


This standard permits different views of the learner information (e.g., learner, teacher, parent, school, 
employer) and addresses issues of privacy and security. The data models associated with this 
specification are not sufficiently complete to cover all the learner data that needs to be exchanged 
between TLA components, particularly those related to pedagogical/andragogical aspects such as 
defining an “Educational Pathway.” 


5.1.19.4 Airmen Learning Record (ALR) 


The vision of the Air Force Learning Services Ecosystem (AFLSE) is to support a continuum of learning 
that deliberately prepares airmen with the required competencies to meet the challenges of the 21* 
century. The ALR is a comprehensive record of all learning airmen achieve during their careers, including 
educational transcripts, training records, performance reports, and ancillary training transcripts. 


3° https://www.imsglobal.org/accessibility/acclipv1p0/imsacclip_infov1p0.html 


31 http://metadata-standards.org/Document-library/Meeting-reports/SC32WG2/2002-05-Seoul/WG2-SEL- 
042_SC36N0175_papi_learner_core_features.pdf 


C-80 | 2019 Total Learning Architecture Report - Appendix C - DoDAF 


The airman’s learning record is a centralized location to record all learning, whether it occurs in a 
specialized training or education program, on-the-job, or even off-duty. The learning record will enhance 
the ability to analyze the readiness of the Air Force by capturing an airman's knowledge and skills gained 
throughout the Continuum of Learning (training, education, and experiences), documenting progress 
and achievements, and identifying gaps and opportunities for growth tied to mission accomplishment 
from both an enterprise perspective as well as an individual level. 


5.1.19.5 Navigator for Integrated Learning Environments (NILE) 


NILE builds upon an existing commercial product that has been historically targeted at the K-12 
educational domain. They use a “Google Maps” approach to create a recommended learning path for 
learners to take. NILE has a content discovery service that has aggregated millions of educational 
resources into their platform. Recommended learner pathways change based on performance, 
instructor input, or student choices. 


The ADL Initiative is funding the migration from propriety data structures and interface specifications to 
a TLA-enabled ecosystem to better evaluate how TLA specifications and standards scale. NILE includes 
numerous enabling technologies (e.g., content discovery service) that are also appealing to the TLA 
ecosystem of tools and technologies. 


5.1.19.6 Adaptive Instructional Sciences — IEEE (C/LT/AIS) P2247.1 


The purpose of the Adaptive Instructional Systems (AIS) Working Group”? is to investigate the market 
need for standards across a group of instructional system technologies. The output of the working group 
will be one or more “Project Authorization Requests” to the IEEE, to develop specifications and 
standards that improve the interoperability across adaptive tools and technologies. 


5.1.19.7 XAPI Launch 


The xAPI Launch specification®’ is a method of launching xAPI-enabled content that works with online 
learning modules, static HTML files, offline content, or immersive games and simulations. It does not 
require a learner identity, the LRS endpoint, or the Session information for how the events should be 
grouped. Implementation requires a minimal HTTP request and handles the creation and transmittal of 
xAPI data to the LRS on behalf of the Activity. While this specification is mature, it is not widely adopted. 


5.1.19.8 cmi5 


cmi5* is an xAPI-based specification that replicates many of the features and capabilities associated 
with SCORM. cmi5 provides definitions of certain xAPI statements, a launch process, a course structure, 
and runtime communications between an LMS and learning content. cmi5 allows the setup of several 
“global” variables including actor, xAPI endpoint and security token, and the automated communication 
of some xAPI statements. The Launch portion of cmi5 allows developers to avoid “hard coding” the LRS 
information into the LMS. cmi5 could be expanded to support a secure, single sign-on experience by 
decoupling the data models and behaviors from the implementation details. Implementation details 
could be defined as part of the cmi5 launch profile which allows it to be used for web and native 
applications. 


32 http://sites.ieee.org/sagroups-2247-1/ 
33 https://github.com/adInet/xapi-launch 
34 https://github.com/AICC/CMI-5_Spec_Current/blob/quartz/cmi5_spec.md#content_launch 
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5.1.20 Assessment Service 


The nature of assessment in a competency-based educational program is more focused on the learner’s 
ability to demonstrate their understanding of key concepts by having them apply their learned skills in 
different contextual situations. The 2018 TLA Reference Implementation used a variety of traditional 
assessment activities that included multiple choice tests/quizzes, Situational Judgement exercises built 
in Unity3D, mobile applications and live group activities. Future implementations require a common 
approach for communicating competency assertions across TLA components and systems. 


Competency evidence will be aggregated from multiple communities inside and outside the 
organization. Performance indicators, organizational metrics, and other systems/databases in use across 
the enterprise will continually be analyzed and assessed to measure the proficiency of individuals and 
teams within the organization. 


Many of the standards and specifications referenced in this document include rubrics for measuring 
performance (e.g., HPML, CASE, Open Badges, etc.). Each has its own data structure, evaluation metrics, 
and format, and each needs to be considered as part of the TLA assessment service. Common 
vocabularies/taxonomies are also required to ensure the myriad of assessments that might be used in 
the TLA are all speaking the right language and crediting the right learner with the right competencies. 


5.1.20.1 Question & Test Interoperability (QT!) 


The QTI specification®> enables the exchange of item and test content and results data between 
authoring tools, item banks, test construction tools, learning platforms, assessment delivery systems, 
and scoring/analytics engines. The data model is described abstractly using UML to facilitate binding to a 
wide range of data-modeling tools and programming languages. 


To support interchange between systems, the data model also supports an XML binding. The IMS QTI 
specification has been designed to support both interoperability and innovation through the provision of 
well-defined extension points. These extension points can be used to wrap specialized or proprietary 
data in ways that allow it to be used with other test items. QTI 2.2 is very stable and has been built on 
other stable, adopted versions of QTI. QTI 2.2 adoption is strong. TLA Assessment Activities could 
leverage the data models of QTI to gain interoperability. However, QTI was not used in 2019. 


5.1.21 Identity Management 


The FLE requires data exchange across different organizational boundaries and inherently different IT 
enclaves. The 2018 TLA Reference Implementation used Keycloak to manage individual access and 
permissions to all TLA components and activities. Usernames were anonymized through the creation of 
a Universal User ID (UUID) that each component used when communicating data about each learner. 


Most tools and technologies are focused on protecting Personally Identifiable Information (PII) inside 
the organizational firewall; however, there are numerous use cases within the TLA where information 
about a learner needs to be communicated between different organizations. 


3° https://www.imsglobal.org/question/index.html#version2.2 
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This is currently achievable at a minimal level through services like ID.me*° and Login.gov?’. Both follow 
the National Institute of Standards and Technology (NIST) SP 800-63 Digital Identity Guidelines*®. 


5.1. 21.1 The OpenIDConnect/OAuth 


Open ID Connect (OIDC) is an open source consortium industry standard for recognizing ID through third 
party verification. It relies on JSON and the use of encrypted identity tokens. It allows for multiple local 
identities to be reconciled to a master identity. It is part of the OAuth family of standards, OAuth is 
associated with access to resources, enabled by that verified identity. 


5.1.21.2 Privacy and Security for TLA (PS4TLA) 


The PS4TLA”’ research seeks to make recommendations for implementing a Privacy-by-Design model 
where privacy and security strategies are an inherent component of the FLE. Areas of study include user 
data characteristics, output characteristics, data location and ownership, data sharing, and privacy 
support mechanisms. This research will result in a requirements specification for tools and technologies 
that manage a learner’s PII and privacy preferences, while also providing individual learners with the 
knowledge required to enact their own privacy control across TLA activities, systems and services. 


5.1.22 Miscellaneous 
5.1.22.1 Learning Tools Interoperability (LT) 


The IMS Global Learning Consortium’s LTI Specification”® prescribes a way to connect learning 
applications and tools with platforms like LMS, portals and learning object repositories, in a secure, 
standard manner. LTI is comprised of a central core and optional extensions to add optional features 
and functions. The LTI core establishes a secure connection and confirms the tool’s authenticity while 
the extensions add features like the exchange of assignment and grade data between an assessment 
tool and LMS gradebook. While LTI is tool-oriented, the underlying data model includes elements about 
the learner and could lend itself well to a TLA Learner Profile. 


5.1.22.2 Sharable Content Object Reference Model (SCORM) 


SCORM defines a specific way of constructing learning content so that it can be shared across Learning 
Management Systems. SCORM is a set of existing standards and specifications that control how content 
is developed, packaged, described, and delivered across systems. A Sharable Content Object (SCO) is the 
most granular piece of training ina SCORM world. The Content Aggregation Model (CAM) determines 
how a piece of content should be delivered in a physical sense. 


At the core of SCORM packaging is a file titled the “imsmanifest.xml.” This file contains every piece of 
information required by the LMS to import and launch content without human intervention. The SCORM 
run-time specifies how the content communicates with the LMS while the content is playing. There are 
two major components to this communication. First, the content must “find” the LMS. Once found, it 
can communicate through a series of “get” and “set” calls and an associated vocabulary. 


36 https://www.id.me/ 

37 https://login.gov/ 

38 https://pages.nist.gov/800-63-3/ 

39 https://www.youtube.com/watch ?v=opmiFzqfwXo 

4° https://www.imsglobal.org/activity/learning-tools-interoperability 
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SCORM is entrenched in the traditional DL community and is still widely used for managing Online 
Learning. It provides an important capability for the TLA to deliver online content. The SCORM/xAPI 
wrapper provides additional capabilities to enable SCORM courses to report to an LRS. 


5.1.22.3 IEEE Actionable Data Book (ADB) 


ADB is a transformative blend of experiential analytics and rich media, delivered through interactive 
eBook technology. ADB was created as a reference model based solely on open standards. These include 
the International Digital Publishing Forum’s ePub3 specification, HTML5, Bluetooth, xAPI, and W3C 
packaging standards for interactive content. It appears work on this specification is no longer ongoing. 


5.1.22.4 ePub3 


The ePub (or EPUB) specification** defines a distribution and interchange format for digital publications. 
The EPUB format provides a means of representing, packaging and encoding structured and semantically 
enhanced Web content — including HTML, CSS, SVG and other resources — for distribution in a single- 
file container. EPUB has been widely adopted as the format for digital books. With new versions, the 
EPUB format's capabilities can support a wider range of publication requirements, including complex 
layouts, rich media, interactivity, and global typography features. EPUB is expected to be used for a wide 
range of content, including books, magazines, and professional and scientific publications. 


This specification is relevant to different types of content the TLA can support and was used as part of 
the Personalized eBook for Learning (PeBL). EPUB in general has seen major adoption, but the adoption 
of version 3.1 is inconclusive at this time. Rules are specific and will promote interoperability among 
tools that leverage EPUB 3.1. This specification indirectly competes with the ADB specification. However, 
ADB was designed with consideration of the EPUB specification. 


5.1.22.5 IEEE P1589 Augmented Reality Learning Experience Models (ARLEM) 


The purpose of the ARLEM specification’ is to become an overarching integrated conceptual model that 
describes interactions between the physical world, the user, and digital information to establish context 
for Augmented Reality (AR) applications used for learning. It will define the data models, modeling 
languages and their bindings to chosen representation formats (e.g. XML, JSON). The specification has 
not yet been released as a version 1.0 specification, so a thorough review has not been possible. 


A review of draft documentation found that while ARLEM attempts to formulate a standardized 
taxonomy around AR, it doesn’t provide enough vocabulary to support Virtual Reality (VR) or Mixed 
Reality (MR) learning environments, and therefore requires extrapolation. As this specification matures, 
the ADL Initiative will continue to investigate how these learning activities can integrate with the TLA. 


5.2 StdV-2 Standards Forecast 
5.2.1 Objective TLA Specifications and Standards 


The TLA objective system state uses a series of TLA and industry specifications and standards for 
message transport, component registry and learning data metamodels under the aegis of the TLA policy 
framework. Identifying, validating, and modifying these specs and standards is a key goal of the ongoing 
TLA research. 


41 https://www.w3.org/Submission/2017/SUBM-epub31-20170125/ 
“2 http://arlem.cct.brookes.ac.uk/ 
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This Standard View 2 (StdV-2) provides the latest vision of the objective system state policy framework. 
It will be updated as these specifications are evaluated, matured, and standardized. 


5.2.2. Core Data Services 


The TLA data strategy includes four major repositories within the data lake. Each repository should 
follow one of the core metamodels specified in the TLA: 


e Learner Profile — stores user attributes and results and conforms to the Enterprise Learner Profile 
metamodel. The ULP metamodel combines portable credentials, universal identity reconciliation 
and traceability to local data including audits of competency evidence, learner state, and global and 
local learner attribute data. 


e Experience Index — stores activities and content that comprise potential learning experiences. It 
conforms to the LRMI specification as amended for the TLA (LRMI+). 


e Competency Framework — stores the jobs, credentials, and KSAO behaviors and the relationships 
between them that define the purpose or educational alignments of learning. It conforms to the 
IEEE P1484.20.1 Reusable Competency Definition (RCD) object standard. 


e Learner Record Store — stores the xAPI statements from learning record providers and listening 
services within the TLA core that define the context under which learning occurred, or /earner state, 
according to the TLA Master Object Model (IEEE P9274.3.1). The LRS captures paradata, which along 
with learning event records and learner state records are in the Javascript Object Notation (JSON) 
protocol defined for the xAPI (IEEE P9274). 


5.2.3. TLA Policy Framework 


The ADL Initiative’s policy document is the DoDI 1322.26. As the TLA achieves initial operational 
capability, this document should shift to a DoD Directive (DoDD) that prescribes the “shall use” language 
for integrating xAPI (IEEE P9274) into distributed learning solutions. As the TLA adoption matures, other 
types of training such as those under the authority of DoDD 1322.18 (Military Training) , DoDI 1322.27 
(Urban Terrain Training), DoDI 1322.24 (Medical Training) , DoDD 5000.59 (Use of Modeling and 
Simulation for Training) and DoDD 3200.11 (Maintenance of Access to Live Ranges for Training) may 
shift to align with the data standards of the ADL Initiative. 


The TLA policy framework includes subordinate specifications and standards as fungible references to 
the DoDI 1322.26. The first standard is the xAPI described in the following sections. The second is the 
TLA umbrella standard defining the “business rules” for adoption, migration, development, fielding and 
sustainment of learning computational assets and data that comport with the TLA goals and interface 
specifications to fully support the TLA provided capabilities. Under the TLA standard are the TLA 
interface specifications and the TLA reference metamodels (see Table 9) that comprise the TLA ecology. 


The TLA policy framework will include service specific instructions for the adoption of TLA compliant 
components to build enclaves, the assignment of overall roles and permissions within the ecosystem, 
and the definition of data model owners to provide the system governance for data metamodel 
configuration control and sharing of analytics templates and results. 
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Table 9. Summary of TLA Objective Specifications. This table shows specifications selected for use in the TLA. The 
specifications are grouped and listed according to the TLA component it is aligned with. There is an expectation 
that TLA Components and their outputs adhere to the referenced specifications. Supporting vocabularies, rules, and 


software ensure this process. 


TLA Component 


Standard/Specification 


Transport 


Data Store / Registry 


JSON payloads 


Activity Registry LRMI++ ECC Metamodel JSON Experience Index 

Schema.org Vocabularies Experience Index 
Activity Stream xAPI IEEE P9274 HTTP / HTTPS with Experience Index 

JSON payloads 

Master Object Model IEEE JSON Transactional LRS 

P9274.3.1 

Cmi5 JSON Transactional LRS 
Competency Management | RCD JEEE P1484.20.1 HTTP / HTTPS with Competency Framework 


LMS 


O*Net 
Credentials CTDL Credential Authoring | Authoritative LRS 
Language 
IMS Global Open Badge 3.0 HTTPS Authoritative LRS 
Learner Profile Enterprise Learner Record HTTPS Learner Profile 
Metamodel 
Learner Service LTI HTTP Experience Index 
OpenAPI HTTP Experience Index 
Identity Management OpenID Connect (profile of HTTP / HTTPS with Learner Profile 
OAuth2.0) JSON payloads 
PrivacyAPI TBD Learner Profile 
Learner Record Provider- | ePUB3+ JSON Transactional LRS 
eBooks 
Learner Record Provider - | SCORM JSON Transactional LRS 


Learner Record Provider — 
adaptive instruction/ITS 


Adaptive Instructional 
Services P2247.1 


Transactional LRS 


Learner Record Provider — 
VR/AR 


IEEE P1589 Augmented 
Reality Learning Experience 
Models (ARLEM) 


Transactional LRS 


The following sections evaluate selected standards and recommended extensions to support TLA 
requirements. Additional insights show how they complement or compete with related specifications. 


5.2.4 Experience Index 


The Experience Index includes an Activity and Resource**Registry and a Content Registry that store 
metadata about TLA learning activities and the content that can optionally be viewed or experienced 


within the activity. 


43 Resources include specific devices, as well as local and non-local (e.g. WWW, “In the wild”) data resources used 
to support learning (e.g. a YouTube video). This registry supports from within the firewall to Zero Trust Network 
(ZTN) security, integrity and non-repudiation. 
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The approach used in the 2018 Reference Implementation relied heavily on the LRMI specification. It 
also had a single “activity index” which treated every document-context instance as unique. The 2019 
Reference Implementation split these into two separate data stores to simplify configuration 
management and to accommodate for the same content being used in different instructional modalities. 


A slightly modified version of LRMI, included as a DIV-3 metamodel in this architecture document, is still 
the transmittal format for experiences. The modified LRMI includes some new fields and modified 
datatypes. The enclave specific data structures for experiences includes site specific attributes to be 
used as part of local device management and adaptation solutions, but the modified LRMI supports 
transmittal to the Enterprise Course Catalog. 


A subset of activities and content within a given installation will be collected into content sets, which are 
ordered hierarchies of activity-content tuples aligned to a Competency Framework, normally referenced 
to a credential of some type (i.e. a badge awarded for completing a course). The content set data 
structure supports transition from the legacy course formats (e.g., SCORM’s Content Aggregation Model 
or Learning Object Metamodel) to the TLA specific structures. Content sets which have been made 
public are visible to the Enterprise Course Catalog service, which includes a single service registry 
pointing to all active course elements in the various experience indices across the FLE. 


5.2.4.1 Learning Resource Metadata Initiative (LRMI) and Extensions 


The Learning Resource Metadata Initiative (LRMI)** specification is a common metadata framework 
developed by Creative Commons and the Association of Educational Publishers for describing and 
tagging educational resources in web-based instruction and training. 


The LRMI metadata schema was adopted by Schema.org in April 2013, which allows anyone who 
publishes or curates educational content to use LRMI to provide rich, education-specific metadata about 
their resources with the confidence that the metadata will be recognized by major search engines. The 
LRMI 1.1 specification is stable and has seen widespread adoption—its metadata attributes are clearly 
defined, and its usage rules drive interoperability. 


The ADL Initiative has conducted extensive analyses of the common educational metadata standards in 

current use. As legacy standards cannot fully accommodate today’s broad range of learning experiences, 
the 2019 TLA Reference Implementation employs an augmented version of LRMI as a transmittal format 
for exchanging data that federates to the Enterprise Course Catalog. This format includes all of LRMI 1.1 
with supplemental data fields to describe learning experiences (content and activities) and content sets. 


A key part of LRMI 1.1 is the AlignmentObject type. This object, used extensively in 202018, describes an 
alignment between a learning resource and a node in an educational framework. The 2019 Reference 
Implementation more explicitly ties these alignments to a Competency Framework metamodel based on 
the IEEE P1484.20.1 Reusable Competency Definition (RCD) specification, with the job_duty_gig and 
role_persona data elements defining a career arc. This allows the AlignmentObject to better serve its 
intended purpose in the context of the TLA. 


#4 http://Irmi.dublincore.org/ 
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In addition, fields were added to enable more detailed data collection and distribution, support mobile 
and cloud technologies, and incorporate non-traditional learning methods, including serious games, 
augmented/virtual reality, LVC (live, virtual, constructive) simulations, and in-person training activities 
such as classes, seminars, and field exercises. Looking ahead, the ADL Initiative seeks to submit a 
metadata strategy recommendation to the DoD and the Office of Personnel Management (OPM) for use 
in their joint DoD Enterprise Course Catalog and Enterprise Learning Record Repository. The ADL 
Initiative identified four criteria that are critical for future metadata standards: 


e Usability — The metadata must be searchable and filterable and provide the information necessary 
for someone not familiar with it to understand its use. Each field must fulfill at least one of those 
functions. 

e Extensibility — The standard must be able to incorporate future enhancements. 

e Real-time performance — Analytics is a vital component of the future learning ecosystem. The ability 
to work with data in real time is becoming increasingly important to learners, educators, 
administrators, and developers. 

e Human and Machine Readability — The data must simultaneously be suitable for human 
consumption and scalable for use with advanced machine learning techniques. 


The ADL Initiative will continue to solicit feedback and recommendations from its stakeholders and 
other communities of interest to improve these metadata recommendations and implementation 
strategies. 


Once the guidance is finalized, the DADLAC will consider formalizing these standards in policy— 
specifically, DoD Instruction 1322.26. That is, the DADLAC is expected to recommend that any new 
courseware developed across the DoD as well as any major refurbishments to existing courseware 
adhere to forthcoming content metadata guidelines. 


5.2.4.2 Schema.org Vocabularies 


Founded by Google, Microsoft, Yahoo and Yandex, Schema.org® is developed by an open community 
with a mission to create, maintain, and promote schemas for structured data on the internet. Each 
Schema.org item type has its own set of descriptive properties. The broadest item type is Thing, which 
has four properties (name, description, URL, image). More specific types share properties with broader 
types. For example, a Place, Person, or CreativeWork is a more specific type of Thing. 


LRMI’s adoption into Schema.org vocabularies has many benefits. In theory, nearly any Schema.org 
Thing could be a learning resource. Therefore, LRMI addresses those metadata properties that 
distinguish content when it is deliberately used for learning. This was done by adding learning resource 
properties to key root types (e.g., CreativeWork), such as educationalUse and educationalAlignment. 


Amore specific type of CreativeWork is a Course.*° A Course is a sequence of one or more educational 
events and/or other types of CreativeWork that aims to build knowledge, competence, or ability of 
learners. They can be distinct instances that take place at different times or locations or are offered 
through different media or modes of study. 


45 https://schema.org/ 
46 https://schema.org/Course 
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As shown in Figure 24, a learning activity is a Schema.org > CreativeWork > Thing that can be used to 
support multiple forms of alignment between a resource and an educational framework. The 
AlignmentObject type can also be used to distinguish between resources that teach and those that 
assess. This presents the ability to collect a substantial corpus of paradata about how different kinds of 
content apply to different instructional domains and enable new insights on which are more effective. 


alignment ramework name 


Type educational 
Framework 


@ ftarget URL 


_educational target name 
Alignment e 
target description 


Creative Work Alignment Object 


Educational 
Framework 


Figure 24. Alignment Between a Schema.org Creative Work and a Node in an Educational Framework“. The 
2018 TLA Reference Implementation used LRMI’s Alignment Object to reference a Competency Framework that 
provided a structured description of required knowledge, skills, abilities, and their interrelated relationships. 


While Schema.org is visible to the AWS sandbox area, it is not generally accessible to protected Global 
Information Grid systems, particularly at higher levels of classification. The TLA enterprise will have to 
maintain a clone of Schema services, through an integrated DoD Schema Server. 


5.2.5 Activity Streams 


One learner may spend time reading technical articles or writing a blog post while another interacts with 
video content and interactive exercises. An Activity Stream is a list of events generated by individuals, 
groups, applications, or learning activities that provide details about the ongoing experiences to other 
TLA components. 


The types and variety of activities that are used for learning can often be associated with a specific 
delivery modality. Instructor-led classroom training will create one set of instructional activities, while 
serious games and simulations have the potential of generating a completely different set of activities. 
This has potential to have two similarly named activities with two different contexts for how those 
activities are being applied and the experiences they encompass. A common vocabulary is necessary to 
ensure all learning activities across different communities accurately describe the experience. By 
formalizing this vocabulary, a set of attributes and rules about the data is established such as how they 
are stored, retrieved and accessed by other components, systems or activities. 


The different activity stream specifications investigated for inclusion in the TLA are similarly structured. 
Each specification includes serialized data streams that consist of statements about activities. Such 
statements typically involve a subject (the person doing the activity), a verb (what the person is doing), 
and a direct object (what the activity is being done to or with). The subject of an activity is nearly always 
the learner but could foreseeably be an instructor, cohort, or other. 


47 https://blogs.pjjk.net/phil/explaining-the-Irmi-alignment-object/ 
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The direct object of an activity is presented differently depending on its context. Verbs need to conform 
to a common vocabulary. Otherwise different organizations will use different verbs to describe the same 
activity or the same verb to describe different activities. 


5.2.5.1 Experience API (xAPI) IEEE P9274 


The xAPI*8 specification is in the process of becoming a standard through the Institute of Electrical and 
Electronics Engineers — Learning Technology Standards Committee (IEEE-LTSC)*’. The xAPI specifies a 
structure to describe learning experiences and defines how these descriptions can be exchanged 
electronically. The main components of xAPI are the data structure called Statements and the data 
storage/retrieval capability, or LRS. The xAPI specification has stringent requirements on the structure of 
these data and the capabilities of the LRS. The transport is HTTP/HTTPS with JSON payloads. 


Statements are data triples that use an Actor, a Verb, and an Object to describe any experience. This 
core syntax can be augmented with extensions, defined in the xAPI profile. Each statement also includes 
timestamps. Actors and Objects use locally unique, resolvable identifiers. The xAPI Profile Specification? 
IEEE P9274.2.1 offers a common way to express controlled vocabularies across these different mediums, 
provides instructions on the formation of xAPI statements, and describes patterns of xAPI usage that 
provides additional context to a domain, device, or system. The xAPI Profile Specification also adds tools 
to support authoring, management, discovery and/or adoption, including additional data elements and 
properties. 


An LRS is the implementation of the server-side requirements associated with the xAPI specification. The 
LRS is a key component of the xAPI architecture. It is the application interface for storing, accessing, and 
often visualizing the data about learning experiences, activities, and performance. The LRS is essentially 

a Web-service that leverages a REST API for the storage and retrieval of xAPI data. 


A learner’s xAPI data and transcripts stored within one LRS can be extracted and sent to other LRSs, 
enabling learning experiences to follow them from one organization to another. An LRS could be 
optionally integrated with any application such as an LMS, Human Resources system, or it could serve as 
centralized data store in an enterprise learning ecosystem. Third party applications which send or 
retrieve learning activity data will interact with the LRS as the data store for xAPI data. From this 
perspective, an LRS is also required to validate the format of the statements as many of the 
requirements in the xAPI specification are targeted toward the LRS component. 


5.2.5.2 Master Object Model (MOM) IEEE P9274.3.1 


The TLA MOM is an xAPI profile that defines an object life cycle for the learner. This approach facilitates 
better composability for TLA compliant enclaves and federations, because there is no technical “state 
manager” required, rather services respond to messages, irrespective of source, that correspond to a 
learning life cycle that encompasses both deliberate (scheduled) and informal (ad hoc) learning. 


48 https://github.com/adIinet/xAPI-Spec 
4° https://www.tagxapi.org/ 
5° https://github.com/adInet/xapi-profiles 
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The MOM also helps to contextualize the conditions under which the learning events occurred, the 
degree of trust in the evidence generated, and the context along the learning continuum from pedagogy 
to self-regulated learning in which the learning event occurred. These help in customizing the learning 
environment to the learner and allow for “edge systems” in the form of advanced learning devices (e.g. 
intelligent tutors or adaptive/self-regulated learning devices) to advance the federation learner state in 
the same way as a Self-reporting event. 


5.2.5.3 cmi5 


The cmi5* is an xAPI-based specification that replicates many of the features and capabilities associated 
with SCORM. cmi5 provides definitions of certain xAPI statements, a launch process, a course structure, 
and runtime communications between an LMS and learning content. cmi5 allows the setup of several 
“global” variables including Actor, xAPI Endpoint and Security Token, and the automated communication 
of some xAPI statements. The Launch portion of cmi5 allows developers to avoid “hard coding” the LRS 
information into the LMS. cmi5 could be expanded to support a secure, single sign-on experience by 
decoupling the data models and behaviors from the implementation details. Implementation details 
could be defined as part of the cmi5 launch profile which allows it to be used for web and native 
applications. There are known security concerns over the JavaScript credentials and the code is not 
portable without modification. 


To facilitate reuse and migration of legacy SCORM content, cmi5 defines a packaging method and a 
communication protocol consistent with xAPI. The Course Structure File within cmi5 is an XML file that 
contains each of the launchable Assignable Units (AU). Together, these replicate the features of 
SCORM’s CAM. SCORM requires all-inclusive and self-contained content and references, but in contrast, 
cmi5 allows the Course Structure File to reference external content. 


The cmi5 object life cycle approximates the SCORM Run Time Environment (RTE - the SCORM 
implementation of IEEE 1484.11) by specifying verbs within the cmi5 profile. The cmi5 course structure 
format extends the SCORM life cycle by allowing "MoveOn" criteria to be met, which can effectively 
"waive" certain AUs such that the LMS would skip over them in delivery to the learner. Objectives can 
also be tagged to any AU or Blocks. The cmi5 xAPI Profile is stable and expects to fold into the xAPI 
standardization efforts of the IEEE as is. 


5.2.6 Competency Management 


Competency management requires the generation of rich and traceable data about learning 
experiences, how they relate to skill proficiency, and the knowledge, skills, abilities, and behaviors that 
individuals need to do their job. Competencies describe specifically what people need to do to be 
effective in their roles, and it clearly establishes how their roles relate to organizational goals and 
success. Each individual role has its own set of competencies needed to perform the job effectively. 


Competency Based Learning represents a transition from curricula focused on abstract knowledge and 
pursuit of certificates to curricula built around authoritative performance indicators that guide learning 
experiences based on challenges in the workplace that unlock human potential. Proficiency is a complex 
and critical concept that requires relevant, trusted data that can be used as evidence about an 
individual’s mastery against a specific set of competencies. 


51 https://github.com/AICC/CMI-5_Spec_Current/blob/quartz/cmi5_spec.md#content_launch 
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A Competency Framework is a structure for defining the knowledge, skills, abilities and attitudes (or 
other characteristics) required to do a job. Competency Frameworks are widely used in business for 
defining and assessing competencies in both hard and soft skills that jointly define successful job 
performance. There are numerous Competency Frameworks available and numerous specifications that 
drive them. 


5.2.6.1 O*Net 


The Occupational Information Network (O*NET)* is sponsored by the U.S. Department of Labor, 
Employment and Training Administration (USDOL/ETA) to facilitate and maintain a skilled workforce. 
Central to the project is the O*NET database, which contains hundreds of standardized and occupation- 
specific descriptors on almost 1,000 occupations covering the entire U.S. economy. Every occupation 
requires a different mix of knowledge, skills, and abilities, and is performed using a variety of activities 
and tasks. As shown in Figure 24, the O*NET database identifies, defines, describes, and classifies 
occupations through Experience Requirements (Training, Licensing), Worker Requirements (Basic and 
Cross-Functional Skills), Occupation Requirements (Work Activities, Context), Worker Characteristics, 
Occupation Specific Information, and Occupational Characteristics. 


The value of the O*NET database to the TLA is in leveraging the existing resources these government 
sponsored activities have already amassed. Their Military Transition search connects occupations in the 
O*NET database to other classifications systems within the military**. This is accomplished through data 
from the Military Occupational Classification (MOC) system at the Defense Manpower Data Center 
(DMDC). This capability is also available via web services. Other resources include “Careers in the 
Military” and links to Army, Navy, Marine Corps, and Air Force COOL projects. 


5.2.6.2 IEEE P1484.20.1 Reusable Competency Definitions (RCD) Working Group (WG) 20 


The current RCD standard™ defines a data model for describing, referencing, and sharing competency 
definitions. The standard provides a way to formally represent key characteristics of a competency, 
independent of its use in any specific context. It enables interoperability among learning systems dealing 
with competency information by allowing them to refer to common definitions with common meanings. 


This specification enables the storage of a Competency Framework in the form of a Directed Acyclic 
Graph (DAG) where competency objects define the knowledge, skills, conditions and other factors that 
role up into an ability to do a job. The edges between each competency object inform the relationship 
between them and have the potential to identify whether they’re dependent, complimentary, 
conflicting, or other. This provides an extensible mathematical underpinning that accommodates the 
relationships defined in any other foreseeable Competency Framework. 


The Data Model for RCD WG 20°° was put together in 2018 to revise the 2008 standard. The WG 20 will 
take the Credential Ecosystem Mapping Project’s mapping of competencies metadata and update RCDs 
to represent the most common elements that are found in multiple standards addressing competencies 
and Competency Frameworks. The ADL Initiative will monitor this working group’s progress. 


°2 https://www.onetonline.org/ 

53 https://www.onetcenter.org/crosswalks.html 
54 https://ieeexplore.ieee.org/document/4445693 
55 http://sites.ieee.org/sagroups-1484-20-1/ 
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5.2.7. Credentials 


A credential is a testament of qualification and competence issued to an individual by an authoritative 
third party. Examples of credentials include academic diplomas, degrees, certifications, professional 
licenses, or other proof of occupational qualification. Within the TLA, credentials are an exchange 
format by a trusted party that formally encapsulate a set of Competencies. By their nature, they are 
linked to other TLA components such as Competency Frameworks, assessment rubrics, or Talent 
Management Systems. This requires the establishment of attributes and rules that allow TLA 
Components to process and compare credentials in the same, interoperable way, particularly in Learner 
Profiles across instantiations of the TLA. This also requires a common way to describe, store, and access 
credentials in order to make fair comparisons of the achievement that was performed to acquire them. 


5.2.7.1 Credential Transparency Description Language (CTDL) 


CTDL*® is a vocabulary comprised of terms that are useful in making assertions about a Credential and its 
relationships to other entities. CTDL is modeled as a directed graph using RDF for describing data on the 
web. Like an activity stream, the "triple" is the basic grammatical construct in making CTDL assertions 
about "things" and is comprised of three simple components: a subject, a predicate and an object. This 
structure allows us to make simple statements that enable a rich description of credential-related 
resources including credentialing organizations and specific subclasses of credentials such as Degrees, 
Certificates, and Digital Badges. Two super classes called Agent and Credential define families or sets of 
subclasses used throughout the CTDL. The primary classes also include the Condition Profile used to 
define sets of constraints on the credential described by an Assessment Profile, Learning Opportunity 
Profile, or Competency Framework. These are used to express learning goals and outcomes. 


5.2.7.2 Credential Registry 


The Credential Registry®’ is both a repository of information regarding credentials and a set of services 
that make it easier to use that information. The Credential Registry works with CTDL to allow the storage 
of credentials that have been expressed using that specification. 


It also registers credentialing organizations that issue these credentials including universities, colleges, 
schools, professional associations, certification organizations, and more. Since it is only a registry, it only 
stores the description of a credential in the abstract sense and does not include any personal 
information or personally obtained credentials. This information will typically be stored in a Learner 
Profile. As a registry, it does allow users to see what various credentials represent in terms of 
competencies, transfer value, assessment rigor, 3 party approval status and more. 


The Credential Engine project's developers are using DCAP process to create systems that communicate 
all virtually all aspects of credentials. The Technical Advisory Committee promotes collaboration across 
different standardization initiatives that are developing data models, vocabularies, and schemas for 
credentials and Competency Frameworks. The Credential Registry uses technology and CTDL to capture, 
link, update, and share information about credentials so it can be organized and centralized in the 
Registry, made searchable by customized applications and linked-to from anywhere on the open Web. 


5° http://credreg.net/ctdl/handbook 
57 https://www.credentialengine.org/credentialregistry 
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5.2.7.3 Open Badges 


Open Badges” are visual representations of verifiable achievements earned by recipients. Open Badges 
is a technical specification and set of associated open source software designed to enable the creation 
of verifiable credentials across a broad spectrum of learning experiences. The Open Badges standard 
describes a method for packaging information about accomplishments, embedding it into portable 
image files as digital badges, and establishing resources for its validation and verification. Open Badges 
contain detailed metadata about achievements such as who earned a badge, who issued it, and what it 
means. The Open Badges 2.0 specification expands on this capability to include versioning, 
endorsements, and full use of JSON-LD. 


Open Badges are expressed as linked data so that badge resources can be connected and reference by 
other systems. This metadata can also be embedded within the graphic. The Open Badges vocabulary 
defines several data classes used to express achievements. There are three core data classes (Assertions, 
Badge Classes, and Profiles) that define mandatory and optional properties as well as the restrictions on 
the values those properties may take. Each Badge Object may have additional properties in the form of 
an Open Badges Extension, a structure that follows a standard format so users can understand the 
information added to badges. Assertions are representations of an awarded badge, used to share 
information about a badge belonging to an earner. 


5.2.8 Learner Profile 


There are challenges in creating a lifelong learning profile. What elements of the Learner Profile should 
the learner be in control of? How does learner information pass from one organization to another? 
What Authoritative Systems have permissions to write to the learner’s profile? How do we represent 
learner context? Within the context of the future learning ecosystem, it is envisioned that a Learner 
Profile will house data about people across their entire career. Adult learners have distinctive personal 
attributes, knowledge, skills, and competencies gained in different contexts through a process of 
development and growth. A Learner Profile may also include demographic data, student interests, 
learning preferences, descriptions of preferred learning environments, personal skills, existing 
competencies and those that need to be developed, socio-emotional health, and myriad other data. As 
student interests change and they become competent in new areas, the profile will update to reflect the 
latest “state” of the learner. 


5.2.8.1 Enterprise Learner Record Repository (ELRR) 


The ELRR is a topic of interest across industry, academia, and military domains. The T3 Innovation 
Network is working closely with standards organizations to complete a standard mapping within a 
comprehensive learner, worker, military record (LWMR). Specific standards initiatives that they are 
working with include: CEDS, Credential Engine (CTDL/ASN), HR Open Standards, IMS Global 
(OpenBadge3.0/CASE), PESC, and IEEE. Their current efforts have produced four use-cases that cover: 1) 
Signal, Search and Discover; 2) Apply, Screen and Verify, 3) Manage Participation, Completion and 
Transition, and 4) Conduct Performance Analytics. ELRR efforts are also supported by the IEEE 
Integrated Learner Record (ILR) standard initiative. Short-term goals include developing ELRR-centered 
use cases and conducting a requirements analysis to determine necessary ELRR components. 


8 https://www.imsglobal.org/activity/digital-credentials-and-badges 
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5.2.9 Launch Service 


The ability to “launch” learning resources across devices, platforms, operating systems, and browsers is 
a fundamental requirement of the future learning ecosystem. A TLA Launch service will ultimately be 
designed as a plugin-based protocol that can launch activities according to different launch standards 
that are applicable to different systems, platforms, or learning modalities. 


5.2.9.1 Learning Tools Interoperability (LT) 


The IMS Global Learning Consortium’s LTI specification®? prescribes a way to connect learning 
applications and tools with platforms like LMSs, portals and learning object repositories, in a secure and 
standard manner. LTI is comprised of a central core and optional extensions to add optional features 
and functions. 


The LTI core establishes a secure connection and confirms the tool’s authenticity while the extensions 
add features like the exchange of assignment and grade data between an assessment tool and LMS 
gradebook. While LTI is tool-oriented, the underlying data model includes elements about the learner 
and could lend itself well to a TLA Learner Profile. 


The xAPI launch specification™ is a method of launching xAPI-enabled content that works with online 
learning modules, static HTML files, offline content, or immersive serious games and simulations. It does 
not require the identity of the learner, the LRS endpoint, or the “Session” information for how the 
events should be grouped. These are coordinated during the device and activity registration process. 
Implementation requires a minimal HTTP request and handles the creation and transmittal of xAPI data 
to the LRS on behalf of the activity. While this specification is mature, it has not yet been widely 
adopted. 


5.2.9.2 OpenAPI! and REST 


The TLA standard specifies the use of OpenAPI and REST for service-to-service communications. 
Database operations using SQL, ODBC and similar connection protocols should be implemented within 
the corresponding core service. The TLA specifies a REST interface for all data insert and request 
operations, through these standardized services. Using REST allows the programs to communicate with 
human readable formatted text using only HTTP requests. Hypertext is the standard way to 
communicate over the world wide web. The OpenAPI project is an industry consortium that proposed a 
simplified, common way to describe the structure and endpoint specifications of APIs. 


For scalability, TLA recommends use of a publish/subscribe (pub/sub) messaging service. Common 
pub/sub services today include the use of streaming services like Kafka or Azure. Specification of a given 
service provider is not part of the TLA framework. The use of a streaming service for the TLA requires a 
listener to sit on the LRS that receives the given HTTP requests and forward the statements to the LRS 
instead of making a direct connection. This prevents having to develop polling messaging services, as the 
TLA specifies the use of stateless services to allow a fully composable ecosystem to evolve. 


5° https://www.imsglobal.org/activity/learning-tools-interoperability 
6° https://github.com/adInet/xapi-launch 
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5.2.10 Identity Management 


The future learning ecosystem requires data exchange across different organizational boundaries and 
different IT enclaves. The 2018 Reference Implementation used Keycloak to manage individual access 
and permissions for all TLA components. Usernames were anonymized through the creation of a UUID 
that each component used when communicating data about each learner. Most tools and technologies 
are focused on protecting PII inside the organizational firewall; however, there are numerous use cases 
within the TLA where information about a learner’s needs to be communicated between organizations. 


This is currently achievable at a minimal level through services like ID.me®™ and Login.gov®™. ID.me 
provides identify management, authentication and group affiliation verification for numerous 
government and business customers. Login.gov allows users to log into multiple government agencies 
with a single user account. The approaches used to manage these capabilities could be relevant to the 
TLA. Both platforms follow the NIST SP 800-63 Digital Identity Guidelines®. 


5.2.10.1 Privacy and Security for TLA (PS4TLA) 


The PS4TLA™ research seeks to make recommendations for implementing a Privacy by Design model 
where privacy and security strategies are an inherent component of future learning. Areas of study 
include user data characteristics, output characteristics, data location and ownership, data sharing, and 
privacy support mechanisms. This research will result in a requirements specification for tools and 
technologies that manage a learner’s PIl and privacy preferences, while also providing individual 
learners with the knowledge required to enact their own privacy control across TLA activities, systems 
and services. 


6.0 PROJECT VIEWS (PV) 
6.1 PV-1 Project Portfolio Relationships 


The PV-1, as depicted in Figure 25, shows the various research portfolio items in the TLA effort and the 
stakeholders/end users they support. 


6.2 PV-2 Project to Capability Relationships 


The PV-3 shows the overall TLA lines of effort. As shown in Figure 26, the projects in the PV-1 contribute 
to these lines (each of which is tied to an element of the TLA policy framework) as defining specification 
requirements, reducing uncertainty for project requirements, or transitioning to a program-of-record 
that can provide the capability. 


51 https://www.id.me/ 

52 https://login.gov/ 

53 https://pages.nist.gov/800-63-3/ 

54 https://www.youtube.com/watch Pv=opmiFzqfwXo 
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1.0 2019 TLA REFERENCE IMPLEMENTATION 


This document describes the logical, physical, and operational implementation of the 2019 Total 
Learning Architecture (TLA) demonstration. It introduces the interface definitions that form the core of 
the objective TLA policy framework upon which future implementations will be built, as well as full 
descriptions of the commercial items used for the current TLA Reference Implementation. 


The 2018 Reference Implementation used point-to-point communication links for data sharing. A goal 
for 2019 was to replace the point-to-point with a scalable publish/subscribe (Pub/Sub) messaging 
service based on Apache Kafka. Beyond the migration to a Kafka-based data streaming architecture, the 
2019 Reference Implementation work developed the software necessary to test, monitor, and evaluate 
critical TLA services, data formats and standards, with an eye toward performance at-scale ina 
heterogenous environment (i.e. the ”future learning ecosystem”). 


The computational hosting environment is provided by USALearning enabling a 24/7 continuous 
environment for validating TLA concepts in operationally relevant environments. While the 2019 work 
was conducted on the development cluster, the Reference Implementation configuration-management 
pipeline supports a three tiered deployment strategy with a development cluster, a test and evaluation 
cluster, and a production cluster to ensure software is efficiently tested before migrating to the live TLA 
Reference Implementation. 


This appendix includes the major sections of a system/subsystem design description (SSDD) — Data Item 
Description DI-IPSC-81431. It is tailored to provide the additional guidance required to reconstitute the 
Reference Implementation given access to a GitHub repository of component containers. 


1.1 Logical Architecture 


Figure 1 is adapted from the Common Education 
Data Standards (CEDS)' data model for 
understanding the relationship between learning 
data and the processes used to create or manipulate Preference 
those data. This model recognizes that each act of 
learning places a /earner, in an organizational 
context, exposed to a learning resource. The 
intersection of all three represents a learning event. 


Learning Event 


The basic processes of student registration, student 
tracking, content presentation, performance 
tracking, etc. currently provided by a Learning Figure 1. The Learning Event. The Learning Event is 
Management System (LMS) must be present in any the intersection of Learner, Organization, and 
learning environment. But the idea of an LMS asa Learning Resources. 

specific product creates a vendor lock for all 

functions within that product, or suite of products, often developed by the same vendor. It also limits 
the workflow options to those supported by the LMS User Experience, which is derived from the 
“factory model” of learning and the “Shannon/Weaver model” of learning transfer inherent in 
traditional models of content delivery. 


Competency 


1 https://ceds.ed.gov/Default.aspx 
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The value of the TLA is facilitated by decoupling LMS functions from different learning activities, and the 
ability to segregate the management of student and instructor experiences within learning 
environments from the actual act of providing learning. 


The factory model of learning starts with an isotropic curriculum and pushes as many students through 
as possible. The Shannon/Weaver model of communication theory posits a transmitter (teacher) and a 
passive receiver (student) of a message (instruction) using assessments to verify receipt. The TLA 
concept of operations (CONOPS) replaces the “factory model” with a learner-centric integrated supply 
chain model. From the domain of integrated business to business (B2B) enterprise resource 
management systems, the TLA now adopts a core/edge paradigm. The edge systems are the devices 
used to provide learning, which may include traditional LMS, as well as handheld devices, intelligent 
tutors, electronic publications, simulators, and any other evolving learning technologies. 


The core systems include the data and services required to conduct ledgering of the data types that 
pertain to planning and controlling individual learning events. These events represent the intersection of 
learners, resources, and organizations. Core services manage the learner bookkeeping functions, while 
back-end core services manage the virtual network bookkeeping functions necessary to operate ina 
distributed, cloud-based environment. Ancillary functions, like an access portal, data visualization tools, 
adaptive algorithms, and any attached learning device, are edge systems that communicate to core. 
Within the 2019 Implementation, learning content — and the activities that host it — are considered edge 
systems. 


The key enabling technology of this CONOPS is the Experience Application Program Interface (xAPI). The 
use of xAPI decouples learning content delivery from the planning and tracking mechanisms. The xAPI 
specification uses a client-server paradigm of Learning Record Providers (LRP) that generate xAPI 
statements and Learning Record Consumers (LRC) that use them. Learning Activities are always acting as 
Learning Record Providers. 


Static content viewers, a version of Moodle using the USALearning xAPI adapter, eventually PeRLS”, and 
various other learning activities generate xAPI statements capturing learning events. The xAPI 
statements are normalized to “actionable information” that propagates through the core services to 
provide evidence of learner competence and are eventually archived to the Learning Record Store (LRS). 


The Learning Activities conform to the TLA data contracts (specifically xAPI and the TLA Master Object 
Model (MOM), used to normalize data) within the learning ecosystem by acting as boundaries between 
the learner and the core services. The composable arrangement of web-based services, data, and 
devices operating with strongly typed data contracts provides these planning and tracking functions. 


The term “data contract” comes from “Service Oriented Architecture” (SOA) software development and 
represents the requirements of digital (application, session, and transport layer) and semantic 
(presentation layer) interoperability between components, deployed as a “contract” for exchange. In the 
2019 architecture, core service components can be both LRP and LRC, which has some interesting 
impacts on the TLA messaging architecture, explored further in the TLA report. This notional 
configuration of xAPI flow is shown in Figure 2. 


? https://adInet.gov/projects/perls/ 
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The 2019 architecture uses a distributed Pub/Sub type messaging 
areas topology that can similarly scale to many concurrent users. The 
Content specific Pub/Sub technology chosen was Apache Kafka, although there 
are similar commercial capabilities available and the TLA policy 
framework is independent of any specific platform. This change 
allowed the test and evaluation of “learning at scale,” a key objective 
oe ae for 2019. It also facilitated the removal of the singleton state manager 
Service Service of the 2018 Reference Implementation, another key objective for 
2019. Many of the primary components used to support the 2018 
Reference Implementation were refactored in favor of event-based 
= microservices, with the events representing potential learner states 
-_ (i.e., the TLA MOM, described later in section 1.5.3.3). 


LRP/LRC LRP/LRC 


Data Lake 


Since multiple learners can access the system simultaneously, and 
may be at different states, the entire enclave can manage as many 
Figure 2. Notional Arrangement of “learner threads” as the individual microservices can handle 
Leming Record Providers: and collectively, rather than the number of threads a single-state manager 
Consumers in the TLA. it ge 

can handle individually. 


The performance of these microservices can be extended horizontally by cloning the processes on 
multiple server instances using cloud-based technology like Hadoop and dynamic load balancing. This 
was a lesson learned from our work with the Navigator for Integrated Learning Experience (NILE) 
platform, described later in this report. 


In summary, the 2019 TLA Logical Architecture reduces complexity by decoupling the features required 
of any “learning system” into separate core services, core data, back-end services, and edge systems. 
This architectural paradigm provides several benefits: 


e Removes vendor lock for the various components of the overall solution; 

e Supports significant horizontal scalability of services; 

e Preserves a non-repudiable audit trail of performance evidence in context, while maintaining 
performance at scale, by careful federation of data and services; 

e Allows for a variety of learning modalities beyond web-based e-learning, including mobile devices, 
intelligent tutors, simulators, operational systems, and legacy LMS solutions; 

e Is resilient to future changes in the types of learning technology and the data and metadata required 
to support that technology and its educational alignment, by careful partitioning of learning data 
into areas of concern: and 

e Accommodates newer models and theories of learning beyond the factory model for how learners 
and mentors arrange and adapt their own learning environments. 


The core service features are separated into service groups associated with the data structures they 
support. The four data structures in the data lake represent aspects of the learner and their possible 
learning paths. They are informed by the TLA data strategy outlined in the 2019 TLA report. Moreover, 
they represent the DoD operationalization of the elements on the CEDS model in Figure 1: the 
characteristics and accomplishments of the learner (Learner Profile), the organization and its defined 
required competencies (Competency Framework), the instructional and assessment candidate learning 
experiences as resources composed of activities, content, and their intersection with competency 
(Experience Index) , and the artifacts recording the conduct of learning events (Learning Record Store). 
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The 2019 refactoring of the TLA Reference Implementation resulted in the microservice based 
architecture shown in Figure 3. The services layer acts as the bridge between learning devices, other TLA 
components, and shared data stores. Each service exposes the stored data to an application so that 
information can be transformed into other meaningful data used by other Reference Implementation 
components. Each service group includes control logic and user interfaces for a set of functions. The 
data contracts between data and service layers are shown based on the nature of the data exchanged. 
The behavior and functionality of each service is defined and aligned with TLA business functions. Input- 
output data flows are identified and aligned with the required TLA data stores. Data models and 
protocols are defined around candidate TLA standards and specifications. The Service Layer includes: 


e Core Services 

o Competency Management — tracks overall competency state for selected goals and makes 
assertions based on evidence provided via xAPI. An instance of the Competency and Skills 
System (CaSS)? hosted the competency framework definitions for the 2019 implementation. 

o Learning Event Management — replaced the recommender system used as part of the 2018 
Reference Implementation. The 2019 Implementation plans, schedules, tracks, and relays 
learner experience data through an internally developed set of independent microservices, and 
tracks learner state through the MOM instead of controlling the implementation state. 

o Activity Registry and Resource Management — manages learning experience registration. In the 
2019 Reference Implementation, this is managed by an activity and resource registry service 
which manages the Experience Indices, containing metadata for all content in the 2019 content 
database. 


e Back-end Services 

o Network Resource Management — discovers TLA services and verifies that a learning resource is 
available. For the 2019 Reference Implementation, the system handled endpoint assignments 
through repository-wide environment definitions, Docker’, and a globally known registry service 
that allowed services to both retrieve locations of services and register their own endpoints. 

o Identity Management — handles protection of PII, login credentials, and identification. In the 
2019 Reference Implementation this was done primarily by Keycloak®, an open-source identity 
and access management solution. The 2019 Implementation also includes two ID-related 
additional services: an xAPI adapter to relay session events, and an authorization endpoint for 
the prototype TLA alias resolution system. 


e Edge Systems 

o Portal — displays basic data and provides a redirect service for the otherwise protected-access 
user interfaces native to each of the services listed above. 

o Decision Management — is based on the Data Analytics and Visualization Environment (DAVE) 
project currently being developed under the ADL Initiative. 

o Learning Devices — are any activities and content that provide learning opportunities and 
generate xAPI statements. For 2019 this includes the Moodle LMS, the Personal eBooks for 
Learning (PEBL), the Pervasive Learning System (PERLS), and the NILE project. 


3 https://cassproject.org/ 
4 https://www.docker.com/resources/what-container 
5 https://www.keycloak.org/index.html 
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Figure 3. Logical TLA Services Architecture. The TLA defines an architectural pattern of multiple services, microservices, and interfaces. Different TLA-compliant 
instances, such as the ADL Initiative’s Reference Implementation, may have arrangements of components that don’t exactly match their logical counterparts, 

depending on the vendors chosen. Communication and coordination between services is vital for a successful realization of the architecture but is not defined or 
constrained as part of the TLA policy framework. 
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1.2 Hardware Architecture 


The 2019 TLA Reference Implementation uses eight virtual machines, listed in Table 1, hosted on 
Amazon Web Services (AWS). AWS provides the back-end platform hosting, virtualization, and Domain 
Name Service (DNS) resolution functions. Each machine was procured under contract to USALearning 
and maintained by the ADL Initiative. The server instances communicate between themselves using 
either HTTP/S over TCP/IP or by producing and consuming messages to the centralized Kafka cluster, 
internally to the AWS campus. External clients accessing the portal, the hosted content, or the service 
redirects may be located outside the AWS campus and connect via REST. The application ports and 
protocols used to access each service are listed in Table 2. 


Table 1. TLA Reference Implementation Server Provisions. Computing and storage presets for each machine used 
during the 2019 Implementation. 


2019 TLA Sandbox 


Primary Component EC2 Type Operating System Volumes | Volume Type | Storage 
CaSS T3.XLARGE UBUNTU 18.04 1 SSD (gp2) 20 GB 
MOODLE T3.XLARGE UBUNTU 16.04 1 SSD (gp2) 8 GB 
MOODLE DB DB.M4.XLARGE UBUNTU 16.04 1 SSD (gp2) 250 GB 
CONTENT T3. MEDIUM UBUNTU 16.04 1 SSD (gp2) 50 GB 
LRS T3.XLARGE UBUNTU 16.04 1 SSD (gp2) 50 GB 
AUTH T3.LARGE UBUNTU 16.04 1 SSD (gp2) 8 GB 
KAFKA M4.LARGE UBUNTU 16.04 1 SSD (gp2) 50 GB 
ACTIVITY REGISTRY T2.MEDIUM UBUNTU 16.04 1 SSD (gp2) 100 GB 
Table 2. Service, Container, and Port Details. Service, port usage, and container layouts of each machine. 
Auth Server https://tla-dev-auth.usalearning.net 


Container 


Service 


Container 
Port 


Public 
Port 


Public 
Path 


Description 


Keycloak Keycloak 8080 proxied auth Learning Experience Index core 
service and UI 

Postgres Postgres 5432 Keycloak's database 

Service Registry Service 8085 proxied registry | Allows admin and self-registration of 

Registry service endpoints 

Nginx Nginx 80/443 80/443 Reverse proxy handling ports / SSL 

Certbot Certbot SSL certificate management and 
automation 


Content Server 
Container 


Container 
Port 


https://tla-dev-content.usalearning.net 
Service 


Public 
Port 


Public 
Path 


Description 


NodeJS Static Server | Video 3000 proxied video Simple xAPI-enabled video player 
Player 

NodeJS Static Server | PDF Viewer | 3000 proxied pdf Simple xAPI-enabled PDF viewer 

Apache Static Apache 80 proxied content | Serves larger static content files 

PALMs PALMs 80 proxied palms ADL's PALMs service 

MySQL MySQL 3306 MySQL DB for PALMs 

PALMs xAPI PALMs DB Monitor sending xAPI about 
XAPI PALMs 

Nginx Nginx 80/443 80/443 Reverse proxy handling ports / SSL 

Certbot Certbot SSL certificate management and 

automation 
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LRS Server 
Container 


https://tla-dev-Irs.usalearning.net 


Service 


Container 


Public 


Description 


Port 


Port 


ADL LRS ADL LRS 8000 proxied root Service running the vanilla ADL LRS 
w/ Django 
Postgres Postgres 5432 LRS's database 
Rabbit MQ Rabbit MQ | 5671 Messaging system used by ADLLRS 
for activities 
Kafka Proxy 1 Kafka 8085 proxied Intercepts xAPI statements, writes 
Proxy them to Kafka topic 
Kafka Proxy 2 Kafka 8085 proxied [ 
Proxy 
Kafka Proxy 3 Kafka 8085 proxied : 
Proxy 
Nginx Nginx 80/443 80/443 Diverts xAPI traffic to proxies, all else 
to LRS directly 
Certbot Certbot SSL certificate management and 


automation 


Container 


Moodle Server 


Service 


Container 


https://tla-dev-Moodle.usalearning.net 


Public 


Public 


Description 


Port Port Path 
Moodle proxied root Moodle instance running through 
Apache 
Nginx 80/443 80/443 Reverse proxy handling ports / SSL 


Note: The Moodle instance uses an AWS RDS instance of Postgres which has been excluded from this table. 


Kafka Server 
Container 


Service 


Container 
Port 


https://tla-dev-kafka.usalearning.net 


Public 
Port 


Public 
Path 


Description 


Kafka Broker 1 Kafka 19092 19092 Single Kafka broker instance 

Kafka Broker 2 Kafka 29092 29092 “ 

Kafka Broker 3 Kafka 39092 39092 : 

Zookeeper 1 Zookeeper | 12181 Single Zookeeper instance 

Zookeeper 2 Zookeeper | 22181 r 

Zookeeper 3 Zookeeper | 32181 = 

Kafka Monitor Kafka 3000 proxied Exposes a protected WebSocket to 
Monitor (HTTP) monitor Kafka topics from a web 

8000 (WS) | 8000 (WS) | monitor | browser or mobile device 

Learning Event Learning 3000 proxied root Responsible for monitoring xAPI 

Manager Event traffic via Kafka and sending 
Manager collateral statements based on what 

it observes 
Nginx Nginx 80/443 80/443 Reverse proxy handling ports / SSL 
Certbot Certbot SSL certificate management and 


automation 
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Metadata Server https://tla-dev-acts.usalearning.net 
Container Service Container Public 10] ) | (mmm DY =X-YoUg fo) d(o) 9} 
Port Port Path 
Experience Index Experience | 3000 proxied | experien | Experience index mapping activities 
Index ce to competencies 
Activity Index Activity 8080 proxied | activities | Activity Index mapping content to 
Index venues / devices 
Activity Registry Activity 3000 proxied | registry | Web Ul and general handler for the 
Registry activity registration 
Postgres Postgres 5432 Learning Experience Index's 
database 
Nginx Nginx 80/443 80/443 Reverse proxy handling ports / SSL 
Certbot Certbot SSL certificate management and 
automation 
CaSS Server https://tla-dev-cass.usalearning.net 


Container Service Container Public 10] ) | (mmm DY SY gfe) d(oy g} 


Port Port Path 
CASS CaSS root Instance of CaSS 
CASS Apache2 Hosts CASS on port 80 
CASS Tomcat Hosts the CASS SkyRepo Database 
CASS SkyRepo 80 CASS’s Database 
Learner Profile Learner 3000 me Stateful representation of a learner’s 
Profile competencies 
Nginx Nginx 80/443 80/443 Reverse proxy handling ports / SSL 
Certbot Certbot SSL certificate management and 
automation 


The 2019 Reference Implementation is installed in an AWS virtual private cloud hosted via contract to 
USALearning. It includes eight virtual machines, which are hosted according to the dynamic load 
balancing that is provided as part of amazon web services Virtual Private Cloud (VPC). AWS provides the 
back-end platform hosting, virtualization, and Domain Name Service (DNS) resolution functions. External 
clients accessing the portal, the hosted content, or the service redirects may be located outside the AWS 
campus. The software components detailed in section 1.3 are installed on servers as shown in Figure 4. 


Learning Activities in 2019 include the USALearning modified version of Moodle and the NILE platform. 
NILE is an open source commercial product developed by Gooru, a Google backed effort dedicated to 
providing alternatives for K-12 training at low cost. The NILE provides an additional learning activity from 
the TLA perspective, as well as a federated LRS. Additionally, as the PeRLS and other learning device 
servers become available, they will provide more connected LRPs. The Generalized Intelligent 
Framework for Tutoring (GIFT) is an intelligent tutoring component that will act as the LRP/activity for 
upcoming work with the Simulation and Training Technology Center (STTC) using the Squad Advanced 
Marksmanship Trainer (SAM-T). The ADL Initiative is also working with the Defense Acquisition 
University to instrument one of their Faculty Professional Development Courses (FPD420) with xAPI, 
which may be connected to the TLA Reference Implementation in 2020. 
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Figure 4. TLA Physical Architecture. This figure outlines the high-level arrangement and flow of information 
between the 2019 Reference Implementation Servers. See Table 2 for server names. 


1.3 Software Architecture 


This section describes the major components and high-level responsibilities for each TLA software 
component, as shown in Figure 5. The TLA Reference Implementation is structured in a way that allows 
the test and evaluation of individual components and capabilities. In practice, many of these 
components will be integrated into a larger system. For example, the learner profile in Figure 5 pulls 
data from both the competency management system and the authoritative LRS to test different 
metamodels, controlled vocabularies, and architectural constraints. Different DoD components already 
have learner profiles planned or implemented as part of their ongoing modernization strategies. The 
Army Training Information System (ATIS) instantiates a learner profile to track individual and units as 
part of the Army Training Management Capability. Likewise, the Air Education and Training Command 
(AETC) is implanting their version of a learner profile within the Airmen Learner Record program which is 
part of the Air Force Learning Services Ecosystem (AFLSE). 
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Figure 5. TLA Component Architecture. Each software “component” refers to a high-level service group that 
typically consist of several smaller services, each “microservice” performing a specialized job for that component. 


1.3.1. Activity and Resource Registry Manager 


An Activity and Resource Registry Manager is the service group associated with capturing, connecting, 
and sharing data about learning resources available to an organization. Key features include the ability 
to generate and manage metadata, taxonomies and ontologies, the alignment of content with 
competencies, paradata, semantic search services, and machine-actionable metadata. In the 2019 
Reference Implementation, the Activity and Resource Registry also catalogs information, usage, 
assertions, and analytical data from a single, sharable metadata repository called the Experience Index. 
This growing collection of data about potential learning resources can be consumed by any TLA 
component. The registry service buffers communication with an enclave’s Learning Experience Index (or 
Indices). The microservices within the 2019 Activity and Resource Manager include: 


e Create, Read, Update and Delete (CRUD) service for Experience Index 
e CRUD Graphical User Interface (GUI) 

e Federation Resource Directory 

e Federation Data compiler 


1.3.2 Experience Index 


The metadata attributes in the Experience Index augment the Learning Resource Metadata Initiative 
(LRMI) specification and makes specific use of the extension for educational alignments. Beyond the 
general metadata content like publisher, name, and location, the team implemented a limited set of 
differentiators for 2019. 


The Experience Index houses the metadata that describes content resources, instructional activities, and 
competency objects (describing educational alignment) that can ultimately link to lessons, courses, and 
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credentials. The Activity and Resource Registry Manager provides a web service to filter and load the 
subordinate lists to semi-automate the creation of these links. The device registry feature for activities 
was greatly simplified for 2019, as a user configurable list (rather than hard coded links as in 2018), but 
this requires further elaboration in 2020 to be user friendly and cybersecurity compliant. 


1.3.3 Activity Providers 


Activity providers are the edge system LRP that represent the boundary between learning technology 
and the TLA core services. Activity providers include LMS servers, simulation hosts, device managers, or 
direct connect devices that host the content, in the form of files, e-publications, scenarios, etc. 
Together, the content and activity form an “experience” which preserves the intent of the learning and 
context under which it occurs. Each LRP is uniquely situated to produce learning records that connect a 
user to learning experiences within an activity. The LRP is responsible for the formatting and population 
of a learning record that meets xAPI requirements and conforms to the TLA MOM. These learning 
records can then be transmitted to the LRS, shown previously in Figure 3. 


1.3.4 Learner Record Store(s) 


xAPI-enabled learning activities generate statements, or records of learning that include a basic triple 
structure consisting of actor, verb, and object. Services transmit these statements over HTTP or HTTPS, 
typically to a central LRS. An LRS serves as a repository for learning records collected from connected 
systems where learning activities are conducted. Initially, the 2019 TLA Reference Implementation used 
a containerized® version of the ADL Initiative’s open source LRS. In late 2019, a more scalable 
commercial solution was configured and implemented. Its main function is to store and retrieve the data 
that are generated from xAPI statements such that all other TLA components may access those data 
without being dependent on direct communication with each other. 


The federated LRS approach separates between noisy LRS (included as edge systems or part of learning 
activities), Transactional LRS, and Authoritative LRS. The Transactional LRS stores all the local state 
messages from the various components generating TLA MOM messages, along with raw evidence and 
baseline assertions. The Authoritative LRS maintains digitally signed verifications of completed 
qualifications, conferred credentials and certifications to perform jobs. In the 2019 Reference 
Implementation they are two logical areas of the same physical LRS. 


1.3.5 Streaming Platform 


During development of the 2018 Reference Implementation, the engineering teams began noticing 
performance and logical issues resulting from services needing to poll an LRS for recent statements. 
Specifically, minor differences in clock-time between servers caused some services to miss statements 
when using a time-gated strategy for polling the LRS. The 2018 solution was to implement an Advanced 
Message Queuing Protocol (AMQP) solution called RabbitMQ’ as a message broker and to configure the 
LRS to forward its new statements to that broker. 


® Containerization is the process of deploying a service using one or more containers. Containers are standalone 
packages of software that contain all necessary dependencies for that software to execute. 


7 https://www.rabbitmq.com/ 
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With this in place, services interested in new xAPI statements could receive them in real-time without 
worrying about timekeeping deltas or polling performance, but the system itself was a reactionary 
afterthought and not incorporated into the data pipeline from the start. 


For the 2019 Reference Implementation, Kafka facilitates the real-time flow of xAPI traffic between 
LRPs, the LRS, and Learning Record Consumers (LRC). Kafka is a horizontally scalable message brokering 
system based on the producer-consumer model, with LRPs as the producers and LRCs as the consumers. 
By incorporating Kafka into the 2019 Implementation, all systems and services could be developed and 
integrated with the expectation of time-stamp correct xAPI traffic as a guarantee. 


Another notable challenge was the xAPI specification’s lack of a mechanism to forward new statements 
directly to interested parties. As a result, the only guaranteed method for extracting xAPI statements 
from an LRS is to request them using filters. The result in 2018 was a choice between losing traceability 
of the human actor that ultimately initiated a chain of events, or a significant reduction in performance. 
In the 2018 data set, both cases were found upon analysis. 


Non-xAPI Traffic 


All xAPI Traffic | 


The proxy is set up as a round-robin 
cluster at the NGINX level. This is to 
offset the issue with it being written in 
NodeJS which is single-threaded. 


Kafka 


LRS 


Figure 6. Apache Kafka Proxy Arrangement. This design enables a “parallel bus” of xAPI statements as described in 
the TLA main report. 


The communication arrangement is shown in Figure 6. To publish the LRS xAPI traffic to the Kafka 
cluster in real time, xAPI traffic passes through a Kafka-integrated proxy, which monitors both the 
original request and the LRS's response. Because the xAPI specification guarantees that the statement's 
MOM-relevant properties are immutable, and because the LRS response contains the assigned Universal 
Unique Identifiers (UUID) for each accepted statement, the proxy constructs a MOM-equivalent version 
of that statement without further LRS interaction. All accepted statements can then be transmitted in 
real-time to the Kafka cluster and subsequently to any component within the TLA network subscribing to 
xAPI traffic. The proxies are physically installed on the same virtual machine as the LRS. 
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1.3.6 Competency Management System 


A Competency Management System manages evidence of an individual’s knowledge, skills, abilities, 
attributes, experiences, personality traits, and motivators to predict their value toward effective 
performance. Competencies might include technical skills, business skills, leadership skills, people skills, 
ethics, professionalism, or any number of topics related to a job. Within the TLA context, the Activity 
Stream(s) stored in the LRS provide the evidence upon which competency assertions are made. The CaSS 
platform was used to manage this evidence and infer competency against a competency framework. For 
the 2019 Reference Implementation, CaSS served as a centralized set of competency frameworks. These 
were mapped to content descriptors to form the activity metadata. CaSS maintains the competency 
framework as part of the main application. In the future, the competency framework(s) will be a 
separate component. The list of Microservices associated with CaSS are: 


e CaSS main 
e = ©Assertion Helper 
e Framework Helper 


1.3.7. Learner Profile 


In the 2019 Reference Implementation, the learner profile is a standalone data service responsible for 
preserving learning records about each user in the enclave and updating that learner’s perceived state. 
The learner profile stores three main types of data: 


e Persistent Learner State Data — A listener in the Learning Event Manager generates the 
goal/task/event relationship for each learner as a function of captured xAPI statements conforming 
to the TLA MOM. This state data must be persistent (stays after browser session terminated) but is 
erased as each task/event is satisfied (based on listening for matching “capture” or “launched” xAPI 
statements). 

e Local and Global Learner Attribute Data — This is minimal for 2019. It stores any characteristics of 
the learner copied from the Enterprise Learning Record Repository (ELRR- a 2020 effort) or entered 
locally for local level assets (such as recommenders). 

e Local Copy of Discoverable Digital Badges — These trace back to the digitally signed conferral and 
assertion statements in the Authoritative LRS and use Linked Data to establish the forensically 
discoverable chain of evidence for learner competency. 


1.3.8 Learning Event Manager (LEM) 


Tracking the lifecycle of a given experience observed in the Activity Stream — most notably the 
attribution of activity in an xAPI statement to an actor’s persistent learner profile — falls under the 
jurisdiction of an LEM. This component consumes xAPI statements from the Kafka stream and relays 
that information to the learner profile and updates any relevant paradata if the statement referred to a 
known activity. The microservices associated with the learning event manager are: 


e Goal Setting: store goal / competency targets for learners 

e Task Scheduling: store scheduled activities for learners 

e Decision Support: store pending requests / status of those requests 

e xAPI Sorting: recognize authoritative xAPI and relay to the authoritative LRS 

e Catalog Listener / Proxy: interface for the known experience index(s) to relay MOM-relevant xAPI 
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1.3.9 Portal/ Single Sign-On (SSO) 


SSO is a property of the TLA’s access control of multiple related, yet independent, software systems. 
Users log in with a single ID and password to gain access to all TLA connected systems. While not a 
formal component of the TLA, it is expected that TLA components will need a centralized auth® system 
to operate within the stakeholder environment. Keycloak was used for this purpose and has been 
integrated to protect Pll and to provide information security for learners, content, and organizations. 
This capability has been integrated to protect Personally Identifiable Information (PII) and to provide 
Operational Security for learners, content, and organizations. 


The SSO capability is integrated with the access portal, and the permissions are relayed to all 
components, preventing the user from having to log in several times. Logically, SSO is part of the back- 
end services (Identity Management), and the portal is an edge system, but they are a single installation 
in the Reference Implementation. 


1.4 Concept of Execution 


For 2019, the workflow for users interfacing with the TLA include a maintenance mode and a learning 
mode. The Maintenance Mode is purely for privileged users (i.e., maintainers) modifying the underlying 
data structures of competency, experiences and learner data. The Learning Mode is for either learners 
or mentors (Observer/Instructor/Controller/Supervisor or OICS) interfacing with the core assets to 
deliberately plan and schedule future learning or interaction with an instrumented device to conduct 
learning. The context under which this interaction occurs supplies insight into the way the learner or 
mentor is configuring their learning environment. Potential contexts are captured in the object life cycle 
of the TLA MOM. This life cycle is based on use cases developed from modern learning theories as 
described in Walcutt & Schatz (2019), Modernizing Learning: Building a Future Ecosystem’. 


1.4.1. Maintenance Mode 


In Maintenance Mode, administrators can access the Activity and Resource Registry. This enables an 
OICS to update competencies, experiences, activities, and content available to learners, as well as 
schedule activities, review curriculum requests, and assign goals. From a component perspective, each 
component responsible for registering and serving metadata in the 2019 TLA Reference Implementation 
reacts to and uses administrator-level credentials from the SSO system and grants special maintenance 
features to those users. For 2019, this mode handled the entry of metadata descriptors for content and 
activities — a control loop labeled Record Management in the service diagram shown on Figure 7. 


1.4.2 Learning Mode 


Figures 8, 9, and 10 show the basic flow for the Learning Mode, which is largely unchanged from the 
learner’s perspective in 2018, albeit with more robust alternate path handling. Learners access a 
common portal, review content relevant to their goals, and receive updated content following the 
system’s experience processing. This mode consists of three primary use cases, each included with a 
corresponding service diagram: goal selection, task selection, and task execution. 


8 The term “auth” is shorthand for authentication, authorization, or both. Keycloak is used for both, with access 
tokens granted by OIDC’s authentication layer which then authorize the user’s current browser session for access 
to Keycloak-protected resources and endpoints. 


° https://bookstore.gpo.gov/products/modernizing-learning-building-future-learning-ecosystem 
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Figure 7. Maintenance Mode. As this mode deals primarily with the registration of metadata and competencies, the information flows all begin with the login 
and portal interactions before moving to either the competency manager or activity registry. The last subroutine deals with the notion of Device Registry, a 
system workflow not built out for the 2019 Reference Implementation but possible using the same components. 
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Figure 8. TLA Learner Planning. The first use case for Learning Mode describes the configuration of a learner’s goals within the system. Through the portal, 
learners can view possible goals known to the competency management component and set those goals for themselves. These goal-setting actions then send 
messages to the LEM which are communicated to the learner’s profile for compilation. Each inner loop typically concludes with information relayed to the LEM, 
which then feeds both the learner profile, LRS, and the activity registry components. Goals themselves are either self-assigned or can be assigned by someone 
operating in Maintenance Mode. Aside from the learner organizing their own goals, an OICS may also assign goals to learners manually through the LEM’s 
administrator controls. These actions trigger similar messages regarding the learner and are then carried across the system, with all goal-setting actions 
prompting a planned statement to the LRS. 
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Figure 9. Learner Task Selection. This phase governs curation and assignment of experiences to a user. Learner task assignments are derived from curated 
content. The incorporation of the LEM enables the system to propose relevant tasks to a learner’s pre-selected goals. The workflow uses the LEM and its Kafka 
incorporation for real-time updates. For learners, this process functions like a recommendation system, with goal-relevant competency associations being 
considered during the curation step. 
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Figure 10. TLA Learner Task Execution. This stage handles the launching and overall activity lifecycle once a learner engages with a selected experience and its 
associated LRPs and work through the corresponding activity. This tracking typically occurs at the LEM component, but the learner profile maintains a user’s 
state and must interpret messages from the LEM to make those state adjustments. It also standardizes the flow of execution as activity streams using TLA MOM 
messages, instead of the ad hoc communication used in 2018. 
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As xAPI statements flow through the LRS, the LEM attributes those actions to the user’s learner profile 
and determines whether these statements satisfy a pending task or requirement requested of the user. 
This task subroutine continues until completion, with the LEM then sending its own xAPI data to close 
the loop. Once an activity is complete, LRP usage data populates paradata for the corresponding 
activities, and the activity registry updates both metadata usage and the learner’s activity preferences. 
Finally, the competency management system receives the most recent evidence and updates all related 
learner state properties. 


1.5 Interface Design 


The TLA is comprised of a collection of open specifications that define the specific message structures 
supported by the application programming interfaces used to expose TLA data, services, and systems. 
The 2019 Reference Implementation interfaces represent the operationalization of service contracts 
defined in the logical architecture. 


1.5.1 Interface Identification and Diagrams 


Communications between 2019 Implementation components involved either HTTPS, WebSocket, or 
Kafka streams. Given the popularity and general simplicity of RESTful APIs (see section 1.5.3 below) in 
modern web services, HTTPS continues to facilitate most traffic within the 2019 Implementation with 
two exceptions: notification of new xAPI statements, and learner profile updates. The primary protocol 
and datatypes are shown in Table 3 below, with fields listed as JavaScript Object Notation (JSON) being 
small datatypes that do not belong to an established specification. 

Table 3. TLA Reference Implementation Protocol and Interface Matrix. Primary protocol and interface usage for 


each component within the 2019 Implementation. For instance, the LRS serves xAPI through its Kafka proxy to three 
services and receives xAPI through HTTPS from five. 


Xl Al AR LP IdM | CF | CM | LRS | LEM | Portal] LA 
Experience (XI) JSON JSON 
Activity Index (Al) LRMI 
Activity Registry (AR) LRMI OIDC XAPI LRMI 
Learner Profile (LP) JSON | JSON 
Identity Manager (IdM) OIDC | OIDC OIDC | OIDC 
Competency Framework (CF) CASS CASS CASS 
Competency Manager (CM) AP xXAPI 
Transactional LRS (LRS) AP AP AP 
Learning Event Manager (LEM) xAPI 
Portal OIDC xXAPI 
Learning Activities (LA) xAPI 

Key: | REST | Kafka 


1.5.2 | Apache Kafka Interface 


Apache Kafka uses a binary protocol over a transmission control protocol connection and defines all APIs 
as request-response message pairs. Clients initiate socket connections with the Kafka cluster, writing 
sequences of request messages and reading back the corresponding responses. 
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1.5.3. Representational State Transition (REST) 


REST is an architectural paradigm that defines a set of constraints to be used for creating web services. 
These constraints maintain data and control logic on the edges of communication and use a series of 
HTTPS-based requests to a Universal Resource Locator (URL) or specified address for a network 
resource. 


1.5.3.1 Experience Application Program Interface 


The xAPI is a specification currently going through a formal standards development process. The main 
components of xAPI are the data structures called Statements and the data storage/retrieval capability 
called the LRS. The xAPI specification has stringent requirements on the structure of data and the LRS 
capabilities. Statements are data triples that use actor, verb, and object to describe any experience. 
Each statement also includes timestamps and unique, resolvable identifiers. The transport is 
HTTP/HTTPS with JSON payloads. 


1.5.3.2 WebSocket 


The WebSocket protocol is a simplified protocol for sending and receiving messages over the Internet, 
abstracting the complexity of TCP handshakes. Rather than polling for updates through HTTP requests, 
WebSocket enables real-time communication between clients and servers with nearly universal 
compatibility. All major browsers support this protocol. 


1.5.3.3 Master Object Model IEEE P9274.3.1 


The TLA Master Object Model (MOM) is an xAPI profile used to define the internal state of TLA core 
systems. It normalizes reporting of learning environment configuration state, learning event state, and 
learner career state so that all attached learning devices report them in the same way. There is no 
overall state controller for the enclave of CORE components. However, the MOM specifies a potential 
sequence of events that trigger each microservice to conduct their data processes. The collective state 
space of the MOM implements a learner “Object Life Cycle” that corresponds to multiple use cases of 
learning. This is explained in detail in the TLA report. 


1.6 Computational Resources 


Part of the systems development plan for the 2019 Reference Implementation was to benchmark 
computational performance. The test appliance included AWS console applications for measuring 
processor loading and memory allocation. Computational testing for 2019 focused on the LRS 
configuration’s tolerance of burst traffic and an analysis of alternatives on technical approaches for the 
upcoming Enterprise Course Catalog and the Enterprise Learner Record Repository projects planned in 
2020 (in collaboration with USALearning and the Office of the DoD’s Chief Management Officer.) 


The initial LRS deployment used the ADL Initiative LRS. The Learning Locker LRS used in 2018 had some 
peculiarities that complicated its integration. Early tests for cybersecurity compliance showed Learning 
Locker had significantly more Category 1 findings than the ADL Initiative LRS. Since cyber lockdowns are 
a principal bottleneck in search performance, the ADL Initiative LRS was determined to be more 
representative of the two. A third LRS from a Data Analytic tools vendor was not available at the 
beginning of the integration period, although this LRS was available and integrated later. 
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Acronym | Definition 


AACRAO American Association of Collegiate Registrars and Admissions Officers 
ADB Actionable Data Book 

ADD Architectural Description Document 

ADL Advanced Distributed Learning 

AEM Adobe Experience Manager 

AEP Association of Educational Publishers 

AETC Air Education and Training Command 

AFLSE Air Force Learning Services Ecosystem 

AFSC Air Force Specialty Code 

AICC Aviation Industry Computer-based Training Committee 
ALIAS Advancing Learning Interoperability Across Systems 
ALR Airmen Learning Record 

AMQP Advanced Message Queuing Protocol 

ANSI American National Standards Institute 

API Application Program Interface 

AR Augmented Reality 

ARCIC Army Capability Integration Center 

ARLEM Augmented Reality Learning Experience Model 

ASN Achievement Standards Network 

ASN-DF ASN Description Framework 

ATC Authority To Connect 

ATD Association of Talent Development 

AU Assignable Unit 

AV All View 

A/V Audio Visual 

AWPAB American Workforce Policy Advisory Board 

AWS Amazon Web Server 

B2B Business to Business 

BIBFRAME Bibliographic Framework 

CAC Common Access Card 

CAM Content Aggregation Model 

CANTRAC Catalog of Naval Training Courses 

CASE Competencies and Academic Standards Exchange 
CaSS Competency and Skills System 

CBL Competency Based Learning 

CCC Common Course Catalog (now Enterprise Course Catalog) 
CD Compact Disk 

CDS Cross Domain Solution 

CDSE Center for Defense Security Education 

CEDS Common Education Data Standards 

CJSE Combined Joint Staff Exercise 

CLR Comprehensive Learner Record 

CMA4LTS Conceptual Model for Learning Technology Standards 
CMI Computer Managed Instruction 

cmi5 Computer Managed Instruction #5 

CMM Capability Maturity Model 

CODIAC Combat Observation and Decision-Making in Irregular and Ambiguous Conflicts 
col Course Of Instruction 

CONOPS Concept of Operations 
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Acronym | Definition 


COOL Credentialing Opportunities On-Line 

CORS Cross Origin Resource Sharing 

CRUD Create, Read, Update, Delete 

CSS Cascading Style Sheet 

CTDL Credential Transparency Description Language 
CV Capability View 

CV Curriculum Vitae 

DACUM Developing A Curriculum 

DADLAC Defense ADL Advisory Committee 

DAG Directed Acyclic Graph 

DAU Defense Acquisition University 

DAVE Data Analytics and Visualization Environment 
DCAM DCMI Abstract Model 

DCMI Dublin Core Metadata Initiative 

DEERS Defense Enrollment Eligibility Reporting System 
DFAS Defense Finance and Accounting Services 

DHA Defense Health Agency 

DISA Defense Information Security Administration 
DIV Data and Information View 

DMDC Defense Manpower Data Center 

DNS Domain Name Service 

DoD Department of Defense 

DoDD DoD Directive 

DoDAF DoD Architectural Framework 

DoDI DoD Instruction 

DSM Design Structure Matrix 

ECC Enterprise Course Catalog 

EDIPI Electronic Data Interchange Personal Identifier 
ELR Enterprise Learner Record 

ELRR Enterprise Learner Record Repository 

FE&T Force Education and Training 

FICAM Federated Identity, Credential, and Access Management 
FIM Federated Identity Management 

FLE Future Learning Ecosystem 

FLUENT Fast Learning from Unlabeled Episodes for Next-Generation Tailoring 
FOC Final Operational Capability 

FRD Functional Requirements Document 

GB Gigabyte 

GIFT Generalized Intelligent Framework for Tutoring 
GIG Global Information Grid 

GUI Graphical User Interface 

HPML Human Performance Markup Language 

HTTP Hypertext Transfer Protocol 

HTTPS Hypertext Transfer Protocol Secure 

I/ITSEC Interservice/Industry Training, Simulation and Education Conference 
IAM Identity and Access Management 

IATC Interim Authority To Connect 

IATT Interim Authority To Test 

IAVM Information Assurance Vulnerability Memorandum 
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ID Identification 

IDA Institute for Defense Analysis 

IEEE Institute of Electrical and Electronics Engineers 
IEEE-SA IEEE Standards Association 

iFEST Innovation, Instruction, Implementation Federal E-learning Science and Technology 
ILR Integrated Learner Record / Interoperable Learner Record 
IMI Interactive Multimedia Instructional 

10C Initial Operational Capability 

IRI Internationalized Resource Identifiers 

ISD Instructional Systems Development (or Design) 

IT Information Technology 

JCIDS Joint Capability Integration and Development System 
JDTA Job, Duty, Task Analysis 

JITT Just-In-Time Teaching 

JSON JavaScript Object Notation 

KSA Knowledge, Skills and Abilities 

KSAO Knowledge, Skills, Abilities, and Other capabilities 
LEM Learning Event Manager 

LIP Learner Information Package 

LMS Learning Management System 

LOM Learning Object Model 

LRC Learning Record Consumer 

LRMI Learning Resource Metadata Initiative 

LRP Learner Record Provider 

LRS Learning Record Store 

LTI Learning Tools Interoperability 

LTSC Learning Technology Standards Committee 

LVCG Live, Virtual, Constructive Game 

LWMCLR Learner, Worker, Military Comprehensive Learner Record 
M&P Manpower and Personnel 

MBSE Model Based Systems Engineering 

MOC Military Occupational Classification 

MOE Measure of Effectiveness 

MOM Master Object Model 

MOP Measure of Performance 

MOS Military Occupational Specialty 

MPTE Manpower, Personnel, Training, and Education 

MR Mixed Reality 

MSEL Master Scenario Events List 

NASAP National Association of Student Affairs Professionals 
NASPA National Association of Student Personnel Administrators 
NATO North Atlantic Treaty Organization 

NCAW National Council for the American Worker 

NDS Normalized Data Schema 

NEC Navy Enlisted Classification 

NILE Navigator for Interoperable Learning Experiences 
NTSA National Training and Simulation Association 

O*NET Occupational Information Network 

OCSTD Occupational Classification Standard 
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ODBC Open Database Connectivity 
OIDC Open ID Connect 
OICS Observer, Instructor, Controller/ Supervisor 
OJT On the Job Training 
OPA Office of Personnel Analysis 
OPM Office of Personnel Management 
OSD Office of the Secretary of Defense 
OUSD(I) Office of the Undersecretary of Defense for Intelligence 
OUSD(P&R) Undersecretary of Defense for Personnel and Readiness 
OV Operational View 
PASTLA Privacy and Security for TLA 
PaaS Platform as a Service 
PAR Project Authorization Requests 
PAPI Learner Public and Private Information for Learners 
PeBL Personalized eBook for Learning 
PEO Program Executive Office 
PERLS PERvasive Learning System 
PESC Postsecondary Educational Standards Council 
Pl Principal Investigator 
Pil Personally Identifiable Information 
PMO Program Management Office 
POI Program Of Instruction 
POR Program Of Record 
PP Pilot Project 
PPI Protected Personal Information 
Pub/Sub Publish / Subscribe 
PV Project View 
QCcL Qualifications/Certifications/Licenses 
QoS Quality of Service 
Qql Question and Test Interoperability 
QRC Quick Response Code 
R&D Research and Development 
RCD Reusable Competency Definition 
RDF Resource Description Framework 
REST Representational State Transfer 
RRL Ready Relevant Learning 
RTS Runtime Service 
SAMR Substitution/Augmentation/Modification/Redefinition 
SAMT Squad Advanced Marksmanship Trainer 
SCO Sharable Content Object 
SCORM Sharable Content Object Reference Model 
SISO Simulation Interoperability Standards Organization 
SOA Service Oriented Architecture 
SOLAR Science Of Learning And Readiness 
SQL Structured Query Language 
SSD Solid State Drive 
SSDD System/Subsystem Design Description 
SSO Single Sign-On 
StdV Standard View 
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STE Synthetic Training Environment 

STIG Security Technical Implementation Guide 
STTC Simulation and Training Technology Center 
SvcV Services View 

SYSML System Modeling Language 

TADLP The Army Distributed Learning Program 
TDT Talent Development Toolkit 

TLA Total Learning Architecture 

TLO Terminal Learning Objective 

TRADOC U.S. Army Training and Doctrine Command 
TRL Technology Readiness Level 

ULR Universal Learner Record (now Enterprise Learner Record) 
UML Unified Modeling Language 

URL Universal Resource Locator 

USDOL U.S. Department of Labor 

UUID Universal Unique Identifier 

VPC Virtual Private Cloud 

VR Virtual Reality 

WAN Wide Area Network 

WAWEF Wide Area Work Flow 

WG Working Group 

xAPI Experience Application Program Interface 
XML Extensible Markup Language 

ZIN Zero Trust Networks 
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